Video, Mobile, and the Open Web

[Also posted at brendaneich.com.]

I wrote The Open Web and Its Adversaries just over five years ago, based on the first SXSW Browser Wars panel (we just had our fifth, it was great — thanks to all who came).

Some history

The little slideshow I presented is in part quaint. WPF/E and Adobe Apollo, remember those? (Either the code names, or the extant renamed products?) The Web has come a long way since 2007.

But other parts of my slideshow are still relevant, in particular the part where Mozilla and Opera committed to an unencumbered <video> element for HTML5:

  • Working with Opera via WHATWG on <video>
    • Unencumbered Ogg Theora decoder in all browsers
    • Ogg Vorbis for <audio>
    • Other formats possible
    • DHTML player controls

We did what we said we would. We fought against the odds. We carried the unencumbered HTML5 <video> torch even when it burned our hands.

We were called naive (no) idealists (yes). We were told that we were rolling a large stone up a tall hill (and how!). We were told that we could never overcome the momentum behind H.264 (possibly true, but Mozilla was not about to give up and pay off the patent rentiers).

Then in 2009 Google announced that it would acquire On2 (completed in 2010), and Opera and Mozilla had a White Knight.

At Google I/O in May 2010, Adobe announced that it would include VP8 (but not all of WebM?) support in an upcoming Flash release.

On January 11, 2011, Mike Jazayeri of Google blogged:

… we are changing Chrome’s HTML5 <video> support to make it consistent with the codecs already supported by the open Chromium project. Specifically, we are supporting the WebM (VP8) and Theora video codecs, and will consider adding support for other high-quality open codecs in the future. Though H.264 plays an important role in video, as our goal is to enable open innovation, support for the codec will be removed and our resources directed towards completely open codec technologies.

These changes will occur in the next couple months….

A followup post three days later confirmed that Chrome would rely on Flash fallback to play H.264 video.

Where we are today

It is now March 2012 and the changes promised by Google and Adobe have not been made.

What’s more, any such changes are irrelevant if made only on desktop Chrome — not on Google’s mobile browsers for Android — because authors typically do not encode twice (once in H.264, once in WebM), they instead write Flash fallback in an <object> tag nested inside the <video> tag. Here’s an example adapted from an Opera developer document:

<video controls poster="video.jpg" width="854" height="480">
 <source src="video.mp4" type="video/mp4">
 <object type="application/x-shockwave-flash" data="player.swf"
         width="854" height="504">
  <param name="allowfullscreen" value="true">
  <param name="allowscriptaccess" value="always">
  <param name="flashvars" value="file=video.mp4">
  <!--[if IE]><param name="movie" value="player.swf"><![endif]-->
  <img src="video.jpg" width="854" height="480" alt="Video">
  <p>Your browser can't play HTML5 video.
 </object>
</video>

The Opera doc nicely carried the unencumbered video torch by including

 <source src="video.webm" type="video/webm">

after the first <source> child in the <video> container (after the first, because of an iOS WebKit bug, the Opera doc said), but most authors do not encode twice and host two versions of their video (yes, you who do are to be commended; please don’t spam my blog with comments, you’re not typical — and YouTube is neither typical nor yet completely transcoded [1]).

Of course the ultimate fallback content could be a link to a video to download and view in a helper app, but that’s not “HTML5 video” and it is user-hostile (profoundly so on mobile). Flash fallback does manage to blend in with HTML5, modulo the loss of expressiveness afforded by DHTML playback controls.

Now, consider carefully where we are today.

Firefox supports only unencumbered formats from Gecko’s <video> implementation. We rely on Flash fallback that authors invariably write, as shown above. Let that sink in: we, Mozilla, rely on Flash to implement H.264 for Firefox users.

Adobe has announced that it will not develop Flash on mobile devices.

In spite of the early 2011 Google blog post, desktop Chrome still supports H.264 from <video>. Even if it were to drop that support, desktop Chrome has a custom patched Flash embedding, so the fallback shown above will work well for almost all users.

Mobile matters most

Android stock browsers (all Android versions), and Chrome on Android 4, all support H.264 from <video>. Given the devices that Android has targeted over its existence, where H.264 hardware decoding is by far the most power-efficient way to decode, how could this be otherwise? Google has to compete with Apple on mobile.

Steve Jobs may have dealt the death-blow to Flash on mobile, but he also championed and invested in H.264, and asserted that “[a]ll video codecs are covered by patents”. Apple sells a lot of H.264-supporting hardware. That hardware in general, and specifically in video playback quality, is the gold standard.

Google is in my opinion not going to ship mobile browsers this year or next that fail to play H.264 content that Apple plays perfectly. Whatever happens in the very long run, Mozilla can’t wait for such an event. Don’t ask Google why they bought On2 but failed to push WebM to the exclusion of H.264 on Android. The question answers itself.

So even if desktop Chrome drops H.264 support, Chrome users almost to a person won’t notice, thanks to Flash fallback. And Apple and Google, along with Microsoft and whomever else might try to gain mobile market share, will continue to ship H.264 support on all their mobile OSes and devices — hardware-implemented H.264, because that uses far less battery than alternative decoders.

Here is a chart of H.264 video in HTML5 content on the Web from MeFeedia:

MeFeedia.com, December 2011

And here are some charts showing the rise of mobile over desktop from The Economist:

The Economist, October 2011

These charts show Bell’s Law of Computer Classes in action. Bell’s Law predicts that the new class of computing devices will replace older ones.

In the face of this shift, Mozilla must advance its mission to serve users above all other agendas, and to keep the Web — including the “Mobile Web” — open, interoperable, and evolving.

What Mozilla is doing

We have successfully launched Boot to Gecko (B2G) and we’re preparing to release a new and improved Firefox for Android, to carry our mission to mobile users.

What should we do about H.264?

Andreas Gal proposes to use OS- and hardware-based H.264 decoding capabilities on Android and B2G. That thread has run to over 240 messages, and spawned some online media coverage.

Some say we should hold out longer for someone (Google? Adobe?) to change something to advance WebM over H.264.

MozillaMemes.tumblr.com/post/19415247873

Remember, dropping H.264 from <video> only on desktop and not on mobile doesn’t matter, because of Flash fallback.

Others say we should hold out indefinitely and by ourselves, rather than integrate OS decoders for encumbered video.

I’ve heard people blame software patents. I hate software patents too, but software isn’t even the issue on mobile. Fairly dedicated DSP hardware takes in bits and puts out pixels. H.264 decoding lives completely in hardware now.

Yes, some hardware also supports WebM decoding, or will soon. Too little, too late for HTML5 <video> as deployed and consumed this year or (for shipping devices) next.

As I wrote in the newsgroup thread, Mozilla has never ignored users or market share. We do not care only about market share, but ignoring usability and market share can easily lead to extinction. Without users our mission is meaningless and our ability to affect the evolution of open standards goes to zero.

Clearly we have principles that prohibit us from abusing users for any end (e.g., by putting ads in Firefox’s user interface to make money to sustain ourselves). But we have never rejected encumbered formats handled by plugins, and OS-dependent H.264 decoding is not different in kind from Flash-dependent H.264 decoding in my view.

We will not require anyone to pay for Firefox. We will not burden our downstream source redistributors with royalty fees. We may have to continue to fall back on Flash on some desktop OSes. I’ll write more when I know more about desktop H.264, specifically on Windows XP.

What I do know for certain is this: H.264 is absolutely required right now to compete on mobile. I do not believe that we can reject H.264 content in Firefox on Android or in B2G and survive the shift to mobile.

Losing a battle is a bitter experience. I won’t sugar-coat this pill. But we must swallow it if we are to succeed in our mobile initiatives. Failure on mobile is too likely to consign Mozilla to decline and irrelevance. So I am fully in favor of Andreas’s proposal.

Our mission continues

Our mission, to promote openness, innovation, and opportunity on the Web, matters more than ever. As I said at SXSW in 2007, it obligates us to develop and promote unencumbered video. We lost one battle, but the war goes on. We will always push for open, unencumbered standards first and foremost.

In particular we must fight to keep WebRTC unencumbered. Mozilla and Opera also lost the earlier skirmish to mandate an unencumbered default format for HTML5 <video>, but WebRTC is a new front in the long war for an open and unencumbered Web.

We are researching downloadable JS decoders via Broadway.js, but fully utilizing parallel and dedicated hardware from JS for battery-friendly decoding is a ways off.

Can we win the long war? I don’t know if we’ll see a final victory, but we must fight on. Patents expire (remember the LZW patent?). They can be invalidated. (Netscape paid to do this to certain obnoxious patents, based on prior art.) They can be worked around. And patent law can be reformed.

Mozilla is here for the long haul. We will never give up, never surrender.

/be

[1] Some points about WebM on YouTube vs. H.264:

  • Google has at best transcoded only about half the videos into WebM. E.g., this YouTube search for “cat” gives ~1.8M results, while the same one for WebM videos gives 704K results.
  • WebM on YouTube is presented only for videos that lack ads, which is a shrinking number on YouTube. Anything monetizable (i.e., popular) has ads and therefore is served as H.264.
  • All this is moot when you consider mobile, since there is no Flash on mobile, and as of yet no WebM hardware, and Apple’s market-leading position.

View full post on Mozilla Hacks – the Web developer blog

Tagged on: , ,

20 thoughts on “Video, Mobile, and the Open Web

  1. JohnSmith

    > That’s true only in a very few countries where such patents are recognised.
    “There are only few people who can’t run ActiveX”. The web, 10+ years ago. Standard reply from supports. Well, I struggled them at my best and looks like we won… only to get backstabbed?! Standards can’t be discriminating. Failing to understand this means all Mozilla mission about to end with EPIC FAIL. If Mozilla is about to allow discrimination based on patents recognition in countries, that’s the very same like discrimination on ability to run ActiveX, etc. After all if I want to publish video, I want it to made universally available to all people, regardles of their country or OS. Not to mention this dirty move puts opensource OSes at disadvantage in countries where patents are recognized.

    > it’s arguable that almost any codec could be covered by such a patent.
    I doubt anyone would dare to attack google. After all they honestly bought the On2 and quite problematic to blame. But anyway, many SW things happens to be developed in those countries. Effectively this would put opensource OSes at competetive disadvantage of force them to pay for licensing H.264. In fact that’s looks like a good backstab to opensource world. Thanks, huh.

    > I’m not sure that the defective systems in a small minority
    > of countries should dictate the approach taken worldwide.
    And how this crappy approach is better than “anything that can’t run activex is a defective, use IE, we don’t support anything else”? Don’t you think it’s the quite the same approach MS took with ActiveX and IE? Everything that is not IE under Windows has been claimed “defective systems”. Unfortunately, Firefox itself has been counted as “defective”. Have Mozilla forgot their hard story and going to backstab those shared their goals?

    > until you face the reality that we were *already* handling H.264 via Flash on desktop,
    Flash is 3rd party addon and sunks slowly. It’s not a part of standards. So I don’t care about it at all.

    > there’s no Flash on mobile,
    And so what? Sure, apple nuts want some patent royalties from h.264. Let’s help them, yep?

    > or shut up.
    Never. I spend enough time on doing bugtracking for FF and promoting FF as I believed I share the same goals as Mozilla does. I’ve been here before even 1.0 appared, using Mozilla Suite and dreaming about web without activeX, web where everyone would be equal, regardless OS, country or whatever, doing my best to file the bugs and promote it. Now looks like Mozilla has abandoned idea that web standards should be non-discriminatory and royalty free. As for me I feel being backstabbed by a traitor when least expected. Pretty cool feeling, ouch!

    As for non-ranting offers: have Mozilla attempted to discuss these matters with google, their teams, etc to clarify google’s position? Maybe there is better way to go, after all?

  2. Brendan Eich

    @JohnSmith: you can call me names and predict Mozilla’s death from behind an anonymous-coward ‘nym, but that’s all empty talk and noise until you face the reality that we were *already* handling H.264 via Flash on desktop, there’s no Flash on mobile, and H.264 in HTML5 video is widespread in Web content sent to mobile devices.

    What is your justification for Mozilla’s long-standing support of Flash to handle H.264? What is your solution to the no-Flash-on-mobile problem? Put up your non-ranting and practical solutions, or shut up.

    /be

  3. Paul Lockett

    “After all, don’t you mind that not each and every system supports H.264 without extra pay?”

    That’s true only in a very few countries where such patents are recognised. The systems in those countries are such a mess that it’s arguable that almost any codec could be covered by such a patent.

    I’m not sure that the defective systems in a small minority of countries should dictate the approach taken worldwide.

  4. JohnSmith

    Well, wars are hard thing and imply some casualities. But as for me it looks like if Mozilla has got their morale low and has abandoned goals of keeping standards free and unencumbered. That’s a worst thing could happen at all. You’re traitor, sir. You’ve backstabbed us. Thank you very much, that’s what we deserve for trying to help make you browser better and promoting it everywhere, doh. Looking on how mozilla develops I can admit it’s jut a poor ripoff of Chrome evolution. But please know: nobody needs “as chrome, but a little worse, little slower, etc”.

    If you’re about to become chrome #2, you already lost your war and will perish. If you’re about to abandon your initial goals and become smth like IE/Chrome, you’re likely unneeded at all. After all, don’t you mind that not each and every system supports H.264 without extra pay? And don’t you mind that this will put ones in disadvantages over others?

    What this reminds me? Well, I can remember old, IE only web. I can remember ActiveX. Can you please explain, how the heck requirement to have patented codec is better than requirement to support activeX? It’s a web breakage and you’ve about to get involved in it. That sucks and there is no excuse for such a blatant abandoning of your initial goals. Shame on you.

  5. Zizzle

    “good computing” experience is subjective.

    Sure you may want the web to turn in a corporate dominated play thing where you have to cough up cash for each click.

    I do not.

    Luckily for you, you can already go an pay-for-play on iTunes and iOS.

    Please leave the open web alone.

    The web only got to where it is because it was open. Don’t try to close it now.

    You cannot ship a free OS or web browser with H.264 support.

  6. Robert Nyman

    When it comes to Windows XP and support: according to statistics, Windows XP has a 45% market share of all desktop operating systems.

    It’s impossible to say how big that will be in the future, but for now it doesn’t seem like a good idea to stop supporting it.

  7. Test

    @sam

    You dream too much. If WebM will not take off then i don’t believe anything else will compete with H.264 for at least for few years.

    You obviously like H.264 so be a man and say you don’t have problems with H.264 as standard web video instead of calling other ungrateful because they don’t agree with you.

    If WebM (VP8) will not take off than i believe WebRTC is lost too at least for few years.

    Google should respond because a lot of valid points where made here and be more reliable partner and if that does not happen we probably lost big time. But hay you like H.264 so you won’t have any problem with that.

    The only reason i responded is because you addressed me directly and not because you said anything of value too me.

  8. sam

    btw, regarding Win XP, i hope mozilla stops supporting that in april 2014 once MS stops supporting it with security patches, we could do with the extra man power being used elsewhere. Ask Opera and google to stop supporting their browsers on XP then too otherwise you may be forced to support it in order to not lose users to chrome as i bet google will carry on supporting as they only care about $.

  9. sam

    Do we have a rough date for when RTC video/audio standards will be chosen? Will IE10 have RTC in it or will there not be enough time? We really need IE10 to have Opus support built-in and RTC support built-in if we want Opus to take off as a standard. October 2012 is when win8 will be on sale so will probably go RTM in July/August along with IE10 as a standalone for Win7.

  10. Jason

    I always thought it was a better idea to use whatever the OS or Hardware supports, but if FF could use H264, which other codecs are possibly being considered if FF uses the OS or Hardware capabilities?

  11. sam

    hopefully RTC will use h.264/Opus as h.264 is the best video codec and has hardware decoding in all devices and opus is the best audio codec. There’s no way MS and apple would add support for WebM for this. I’d focus on making sure Opus gets picked as the audio codec.

  12. sam

    I was referring to being greatful of mozilla and also h.264, the video quality is better and the filesize is smaller. We won’t have to pay for the licence as win vista/7 already has h.264 and there is hardware decoding in pretty much all smartphones.

    Web developers can still choose to use webm if they want, mozilla isn’t removing support and i doubt google will either, whether they stop development is another matter entirely.

    Daala could potentially kill HEVC (a.k.a h.265) if the spec is better than theirs, we can easily make our codec faster to encode/decode than theirs depending on the spec of course as we have huge numbers of open source contributors, just look how great x264 is!

    You do a lot of complaining but you don’t offer suggestions, how can mozilla force all mobile platforms to have hardware decoding onboard, how can they add hardware decoding for 99% of phones that don’t currently have it, how can they make the video quality and filesize better instantly, how can they get google to remove h.264 capability. They can’t do any of these things, they are out of their hands. Blame google and adobe.

    Mozilla has learn’t a lot from this mistake, with some careful planning the problems can be prevented from happening with Daala/Opus.

    As for all your comments about Daala replacing h.264, do you really think that in 2yrs time (assuming daala is out by then) that mozilla will stop using h.264 and the whole internet will switch? h.264 will be around for MANY years, they will both be used. Do you really think everyone upgrades their hardware as soon as new hardware is released?

    Mozilla need to get detection of gpu/apu built into the html5 spec that brings up a dialog box asking the user if it is ok to share this info with the website. Then if hardware decoding support for Daala is found the youtube for example can tell the browser to use Daala, if not it will send h.264. Daala will use far more cpu than h.264 therefore it would be a bad idea to send it to anyone who doesn’t have hardware decoding for it. h.264 spec was released in 2003, hardware has been around for around 7yrs, you can’t just kill off h.264 and replace it. People don’t want/need to upgrade to another hardware device to be able to use Daala, they will both be available just like how h.264/webm are now. After 6yrs of daala hardware decoders being available most people will be using devices capable of it therefore maybe then h.264 will be cut loose. Codecs are long-term projects.

    Now developers just need to create 2 .mp4 files, 1 with h264/aac and another with h264/vorbis. they wouldn’t need to encode 2, just mux the audio. This will satisfy all html5 browsers assuming opera follows firefox’s lead.

  13. Wolfgang Spraul

    Let’s build our own hardware, collaboratively and with free licenses. Google has ‘opened’ the webm hardware encoder and decoder sources only behind a longform IP license, i.e. not at all. Elphel has done groundbreaking Ogg Theora hardware encoding over 5 years ago, but nobody cared. Sources are still out there under GPL. Mozilla doesn’t need to shed crocodile tears – read your own blog post and you realize the decision is made ‘in hardware’. So if ‘we will never give up, we never surrender’ should mean more than heroic blog posts, join one of the many great open hardware movements out there, with full power! This stuff is easier than you may think, no need to treat ‘hardware’ as that magic black box where all the decisions you don’t like somehow get made…
    Checkout http://www.milkymist.org for a GPL licensed SoC. No video hardware decoding and encoding. Why not? We are working on it – join if you like.
    Cheers, keep it up!
    Wolfgang

  14. Test

    “Yes, we still support WebM. We’d like to see it both evangelized harder and improved technically.”

    I agree.

    “There may be work I’m not aware of to do both going on at Google, but can anyone tell?”

    I can’t confirm but my gut tells me they are.

  15. chico sajovic

    A lot of people’s principals are getting in the way of good a computing experience. Why do people care about the H.264 license? H.264 is a good product, it has a lot of OS support, it has a lot of hardware support. Hopefully we can move on into a world where all video is H.264. Pay the fee if you can or use OS level support if you can’t. The end goal is not free but one high quality format supported by all producers and all consumers.

  16. Brendan Eich

    Yes, we still support WebM. We’d like to see it both evangelized harder and improved technically. There may be work I’m not aware of to do both going on at Google, but can anyone tell? It’s not affecting Android or Chrome on ICS.

    To clarify talk about Xiph, work on Opus, Daala, and future stuff: we have to keep doing this. It’s necessary since the state of the encumbered art keeps evolving. Fighting the next battle includes doing new formats and codecs (and IPR protection for them, not just from deep pockets but at least from big partners).

    Winning in the long run consists of having better unencumbered tech. H.264 will die hard but there will be new and better formats. See my point in an earlier comment about WebRTC needing low-latency encoding.

    /be

  17. Test

    “WebM does not have hardware decoding on too many devices. It’s coming but too late.”

    I do agree more could be done yes but on other hand in two years time i think something has been done.

    “I heard Google tried to require it of Android OEMs and they said “no”. That’s the rumor.”

    “and many devs willing or able to afford dual-encoding”

    But then we lost? Then we can win because the answer will always be no if it’s not H.264?

    “Apple market primacy.”

    But Android has grater market share and this should kick start WebM on Android devices?

    “Your last paragraph asks if there is anything more.”

    But there is nothing i should know about WebM that would change my support toward it? Mozilla still does believe in WebM but you would like Google would push harder?

Leave a Reply