Video Processing at Dropbox

A description of HLS encoding workflow at Dropbox.

HTML5 Video at Netflix

Today, we’re excited to talk about proposed extensions to HTML5 video that enable playback of premium video content on the web. We currently use Microsoft Silverlight to deliver streaming video to web browsers on the PC and Mac. It provides a high-quality streaming experience and lets us easily experiment with improvements to our adaptive streaming algorithms.

But since Microsoft announced the end of life of Silverlight 5 in 2021, we need to find a replacement some time within the next 8 years. We'd like to share some progress we've made towards our goal of moving to HTML5 video.

Silverlight and Browser Plugins
Silverlight is a browser plugin which allows our customers to simply click "Play" on the Netflix website and watch their favorite movies or TV shows, but browser plugins have a few disadvantages. First, customers need to install the browser plugin on their computer prior to streaming video. For some customers, Netflix might be the only service they use which requires the Silverlight browser plugin. Second, some view browser plugins as a security and privacy risk and choose not to install them or use tools to disable them. Third, not all browsers support plugins (eg: Safari on iOS, Internet Explorer in Metro mode on Windows 8), so the ability to use them across a wide range of devices and browsers is becoming increasingly limited. We're interested to solve these problems as we move to our next generation of video playback on the web.

HTML5 Premium Video Extensions
Over the last year, we've been collaborating with other industry leaders on three W3C initiatives which are positioned to solve this problem of playing premium video content directly in the browser without the need for browser plugins such as Silverlight. We call these, collectively, the "HTML5 Premium Video Extensions":

Media Source Extensions (MSE)
The W3C Media Source Extensions specification "extends HTMLMediaElement to allow JavaScript to generate media streams for playback." This makes it possible for Netflix to download audio and video content from our content delivery networks and feed it into the video tag for playback. Since we can control how to download the audio/video content in our JavaScript code, we can choose the best HTTP server to use for content delivery based on real-time information, and we can implement critical behavior like failing over to alternate servers in the event of an interruption in content delivery. In addition, this allows us to implement our industry-leading adaptive streaming algorithms (real-time selection of audio/video bitrates based on available bandwidth and other factors) in our JavaScript code. Perhaps best of all, we can iterate on and improve our content delivery and adaptive streaming algorithms in JavaScript as our business needs change and as we continue to experiment.

Encrypted Media Extensions (EME)
The W3C Encrypted Media Extensions specification "extends HTMLMediaElement providing APIs to control playback of protected content." The video content we stream to customers is protected with Digital Rights Management (DRM). This is a requirement for any premium subscription video service. The Encrypted Media Extensions allow us to play protected video content in the browser by providing a standardized way for DRM systems to be used with the media element. For example, the specification identifies an encrypted stream format (Common Encryption for the ISO file format, using AES-128 counter mode) and defines how the DRM license challenge/response is handled, both in ways that are independent of any particular DRM. We need to continue to use DRM whether we use a browser plugin or the HTML5 media element, and these extensions make it possible for us to integrate with a variety of DRM systems that may be used by the browser.

Web Cryptography API (WebCrypto)
The W3C Web Cryptography API specification defines an API for "basic cryptographic operations in web applications, such as hashing, signature generation and verification, and encryption and decryption." This API allows Netflix to encrypt and decrypt communication between our JavaScript and the Netflix servers. This is required to protect user data from inspection and tampering, and allows us to provide our subscription video service on the web.

First Implementation in Chrome OS
We've been working with Google to implement support for the HTML5 Premium Video Extensions in the Chrome browser, and we've just started using this technology on the Samsung ARM-Based Chromebook. Our player on this Chromebook device uses the Media Source Extensions and Encrypted Media Extensions to adaptively stream protected content. WebCrypto hasn't been implemented in Chrome yet, so we're using a Netflix-developed PPAPI (Pepper Plugin API) plugin which provides these cryptographic operations for now. We will remove this last remaining browser plugin as soon as WebCrypto is available directly in the Chrome browser. At that point, we can begin testing our new HTML5 video player on Windows and OS X.

We're excited about the future of premium video playback on the web, and we look forward to the day that these Premium Video Extensions are implemented in all browsers!

By Anthony Park and Mark Watson, The Netflix Tech Blog

The State of Streaming Media Protocols 2013

Protocols are a geeky thing, almost up there with metadata or IP address schemes. Without protocols, there would be no concept of a webpage, email delivery, or even VoIP (voice or video over IP). In fact, there would be no streaming of any flavor.

With that in mind, let's take a quick look at the latest developments in a few key protocols, some of which are just coming into widespread use for streaming.

On Time or On Target? The UDP Question
You may have heard the truism that "time is money." We often use that concept when we talk about hardware- or software-based transcoding: If a job needs to be done quickly, even faster than real time, go for the hardware; if time is less critical, use software.

The same concept, slightly morphed, can be applied to the world of streaming protocols: If low latency is key, go with UDP (User Datagram Protocol), but if delivery guarantee is more critical, go with TCP (Transmission Control Protocol). The latter provides control over guaranteed packet delivery -- the control in the transmission control protocol name -- while the former does not.

Every other protocol that we discuss in this article will hinge on TCP, but recent advances on the UDP front bear mention.

I'm not going to retrace step by step the key points that Dom Robinson travels in his recent article titled Reliable UDP (RUDP): The Next Big Streaming Protocol?, but I do want to point out several highlights.

First, in and of itself UDP is unreliable, but it's fast and efficient. The protocol itself has no mechanism to guarantee delivery or request missing packets, as does TCP. A properly constructed application, though, can act as the first line of defense in detecting loss or packet corruption with subsequent request for dropped packets.

"It can take TCP upward of 3 seconds to renegotiate for the sequence to restart from the missing point," wrote Robinson, "discarding all the subsequent data, which must be requeued to be sent again. Just one lost packet can cause an entire ‘window' of TCP data to be re-sent."

Second, UDP can work hand-in-hand with error-correction techniques. Many legacy intermittent networks, including those based on asynchronous transfer protocols such as ATM, used Forward Error Correction (FEC) to anticipate intermittent outages. FEC provides "packet flooding" at a certain percentage above 100% of packets -- be it 15%, 20%, 30% -- to then allow the client application to reconstruct the missing or corrupt packets without the need to request that packets be retransmitted.

Third, the concept of reliable UDP has been around for quite some time and can be accomplished with a number of open source tools, but there are also a number of commercial licensees that offer tools based on RUDP.

Your mileage may vary if you want to tinker under the hood of a freeware application such as UDP Data Transfer, or you may just want to reach out to a vendor in the RUDP space. Regardless of your choice, there are enough RUDP options out there to warrant research, especially if very low latencies and FEC are part of your workflow.

HTTP Über Alles
The fanfare around Dynamic Adaptive Streaming over HTTP, or DASH for short, continues to build at a rate we'd only ever seen back in the early streaming days of Real versus Microsoft. DASH was ratified back in late 2011, but its adoption has moved at a rapid pace.

Let's take a look at the HTTP flavors out there in current circulation: HDS, HLS, MPEG DASH, and Smooth Streaming. We'll look first at DASH followed, in alphabetical order, by the Adobe, Apple, and Microsoft proprietary offerings.

DASH It All, or Just 264?
Due to DASH's ability to deliver any of several types of files -- from the fragmented MP4 version of ISO Base Media File Format (ISOBMFF) to the Apple-modified MPEG2 Transport Stream (M2TS) -- the DASH specification reads like an encyclopedia of encoding, encryption, and delivery technologies.

To offset the potential problem that plagued MPEG 4 system -- a wide-ranging specification, based on Apple QuickTime, that was too complex to easily implement -- the industry has begun looking at options and commonalities of all the major HTTP delivery solutions. It has, as of the time of this writing, begun drafting a subset specification that centers on H.264 served as fragmented MP4. This use of ISOBMFF and H.264 is being dubbed as DASH 264.

What impact the DASH 264 specification has on the potential of bringing Apple into the DASH fold -- it was a contributor to the DASH specification, but is the only M2TS-based HTTP solution -- remains to be seen. But several industry players we spoke to stated the strong need to get solid DASH implementations into the field in early 2013.

Adobe's Flash vs. DASH Conundrum
While Adobe wouldn't publicly come out in support of DASH, despite co-sponsoring an ISOBMFF white paper in late 2011, the company did put its weight behind DASH in late February 2012.

"I am excited to announce that Adobe's video solutions will adopt the emerging video standard, MPEG-DASH across our video streaming, playback, protection and monetization technologies," wrote Kevin Towes in an Adobe blog post. "Adobe will support MPEG-DASH ISOBFF on demand and live profiles which are recommended in DASH-264 recommendation."

Towes anticipated the question many would ask: What about Flash and the Adobe HTTP Dynamic Streaming (HDS) protocol? He noted that Adobe will continue to push forward with HDS, even as Adobe supports DASH.

"Adobe will continue developing its HDS format used to deliver high quality, protected video experiences across multiple devices," wrote Towes, noting that the DASH profile for ISOBMFF "is similar to Adobe's HDS format and supports many of the performance objectives of the HDS format."

The continued development of HDS makes sense, as it offers Adobe the chance to try out new functionality within the confines of the Flash Player, but as we move into 2013, we wonder whether this is a sustainable model.

Adobe MAX 2013, to be held in May, may shed some light on HDS -- and perhaps even RTMP -- but meanwhile Adobe continues to show DASH functionality in the Flash Player.

An Apple (Protocol) a Day
Much has been made of the idea that HTTP Live Streaming (HLS) is a standard in the marketplace, but as one panelist at a recent DASH event quipped, "The IETF drafts of the Pantos spec are more a suggestion than a standard."

The Pantos spec, as it is known in the industry, is a series of working drafts for HLS submitted by two Apple employees as an information draft for the Internet Engineering Task Force. As of the time of this article, the Pantos spec is currently at informational version 10.

Much has changed between the early versions and the most recent v10 draft, but one constant remains: HLS is based on the MPEG-2 Transport Streams (M2TS), has been in use for almost 2 decades, and is deployed widely for varied broadcast and physical media delivery solutions.

In that time frame, however, little has changed for basic M2TS transport stream capabilities. For instance, M2TS still lacks an integrated solution for Digital Rights Management (DRM). As such, all HLS versions cannot use "plain vanilla" M2TS, and even the modified M2TS used by Apple lacks timed-text or closed-captioning features found in more recent fragmented elementary stream streaming formats.

Yet Apple has been making strides in addressing the shortcomings of both M2TS and the early versions of HLS: In recent drafts, the HLS informational draft allows for the use of elementary streams, which are segmented at the time of demand rather than beforehand. This use of elementary streams means that one Achilles' heel of HLS -- the need to store thousands, tens of thousands, or hundreds of thousands of small segments of long-form streaming content -- is now eliminated.

Google, with its Android mobile operating system platform, has adopted HLS for Android OS 4. Some enterprising companies have even gone back and created HLS playback for earlier versions of Android OS-based devices.

Smooth Streaming Ahead?
No discussion of HTTP protocol-based streaming delivery would be complete without a mention of Microsoft Smooth Streaming. After all, the Protected Interoperable File Format (PIFF) is the basis for the Common File Format (CFF) that is being used for UltraViolet, and the Common Encryption Scheme (CES) is based partly on Microsoft's 2008 idea that common encryptions and encoding could be implemented for use around the fragmented MP4 standard.

As we enter 2013, rationalization of PIFF-CFF-CES and the upcoming Common Streaming Format (CSF) will continue to pare down the number of options, which is a good thing if we are to get back to the business of creating content and delivering it to anyone who wants to view it.

Yet Microsoft isn't resting on its laurels, as the company announced in late 2012 that it would be supporting Smooth Streaming via the Open Source Media Framework (OSMF) that is part of Adobe's Strobe initiative for Flash. Yes, you heard that right: Not only does Flash Player support DASH in beta (thanks to Adobe), but it now supports Smooth Streaming (thanks to Microsoft).

Is RTSP Dead?
One of the most venerable streaming protocols, the Real-Time Streaming Protocol, has been implemented natively into every type of device: set-top boxes, smartphones, tablets, and PCs. Yet these implementations are often fraught with buggy code, limited support, and a number of oddities.

In testing performed on a number of Android OS devices, we have been surprised to find that RTSP-based video playback -- served from a standards-based server -- could not be played with built-in applications and services, despite the requirement for the base Android OS to be able to play this content. Content that would play consistently on numerous RTSP implementations would stop dead in its tracks on other devices.

This was true of any standards-based streaming protocol in the late 1990s, but these days, consumers just expect their content to stream with limited buffering and at varying data rates. RTSP offers neither of these as certainty, but is quite inexpensive to implement, so we suspect that it will be with us for at least a few more years before retiring to greener pastures to make way for DASH and more recent streaming protocols.

Where Does RTMP Fit?
At a recent informational meeting I attended with a major software company, a slide that was shown caught my attention, more for the lack of what was shown than for what the slide contained.

This particular slide listed the typical agnosticism -- codec, protocol, player -- of their soon-to-be-released update, and it had the regular litany of compatibilities. Yet what caught my attention was that the sparsest, by far, was the protocol list.

Only two protocols were noted -- HTTP and RTMP -- so I asked why RTMP was listed when other non-HTTP protocols were not. To summarize their response, they said RTSP and other non-HTTP protocols weren't being requested at all, but RTMP was still a valid industry solution.

Part of the reason lies in the fact that RTMP is "true" streaming with very low latencies and session "statefulness" that can't yet be found in HTTP-based delivery. In addition, RTMP is firmly entrenched on the vast majority of devices -- with the exception of the iOS devices -- thanks to the inclusion of the Flash Player on handsets, tablets, mobile devices, and PCs.

Yet, for all that entrenchment, as we've noted previously, Adobe continues to lean toward the HTTP delivery model in all the important ways including monetization functionality. We're not ruling out RTMP, but we do understand that the scalability and interoperability of HTTP solutions such as MPEG DASH and HLS offer compelling reasons to surf the fine streaming waves coming out of Apache servers everywhere.

Conclusion
So what does 2013 hold? We think the year is DASH's to lose.

Once DASH is officially supported in Flash, we see the possibility that DASH will be firmly enough entrenched to begin "hockey stick" growth for online video delivery. If DASH 264 can be implemented as quickly as it appears it will be ratified, and if there is some form of rationalization between HLS and DASH, including the ability to include Apple's DRM scheme in the Common Encryption Scheme, we might just note 2013 not only as the beginning of true online video delivery growth but also as the point at which cable and satellite providers begin to pay attention to delivery to all devices -- including set-top boxes -- for a true TV Everywhere experience.

By Tim Siglin, StreamingMedia

Streaming Adaptive - a Brief Tutorial

An interesting white paper by Niels Laukens (EBU).

New Data Released on the Performance of Adaptive Streaming over HTTP

Adaptive streaming over HTTP is gradually being adopted by content owners as it offers significant advantages in terms of both user quality and resource utilization for content and network service providers. To understand whether today's existing commercial players perform well, especially under dynamic network conditions, Cisco and The Georgia Institute of Technology just released a technical white paper on the subject.

Their experiments covered three important operating conditions:
- First, how does an adaptive video player react to either persistent or short-term changes in the underlying network available bandwidth? Can the player quickly converge to the maximum sustainable bitrate?

- Second, what happens when two adaptive video players compete for available bandwidth in the bottleneck link? Can they share the resources in a stable and fair manner?

- And third, how does adaptive streaming perform with live content? Is the player able to sustain a short playback delay?

The paper identifies major differences between Microsoft's Smooth Streaming, the player used by Netflix, and one open source player (Adobe OSMF). Their findings report that the Smooth Streaming player is quite effective under unrestricted available bandwidth as well as under persistent available bandwidth variations. It quickly converges to the highest sustainable bitrate, while it accumulates at the same time a large playback buffer requesting new fragments (sequentially) at the highest possible bitrate.

On the negative side, the paper says that the Smooth Streaming player reacts to short-term available bandwidth spikes too late and for too long, causing either sudden drops in the playback buffer or unnecessary bitrate reductions. Further, the experiments with two competing Smooth Streaming players indicate that the rate-adaptation logic is not able to avoid oscillations, and it does not aim to reduce unfairness in bandwidth sharing.

The Netflix player is similar to Smooth Streaming as they both use Silverlight for the media representation. However, the paper reports that the former showed some important differences in its rate-adaptation behavior, becoming more aggressive than the latter and aiming to provide the highest possible video quality, even at the expense of additional bitrate changes. Specifically, the Netflix player accumulates a very large buffer (up to few minutes), it downloads large chunks of audio in advance of the video stream, and it occasionally switches to higher bitrates than the available bandwidth as long as the playback buffer is almost full. It shares, however, the previous shortcomings of Smooth Streaming.

The paper says that the OSMF player often fails to converge to an appropriate bitrate even after the available bandwidth has stabilized. This player has been made available so that developers will customize the code including the rate-adaptation algorithm for HTTP Dynamic Streaming for their use case.

By Dan Rayburn, Business of Video

The Impact of Adaptive Rate Streaming

An interesting white paper by Benjamin Schwarz (Verimatrix and Harmonic).

Orange Ready for the Next Big Challenges

Having established itself as a major Pay TV provider and pioneered multi-screen TV, Orange provides one of the best examples of how a telco can transform itself from a voice/data provider to a media company. The France Telecom brand also demonstrates the impact that video-over-IP has had on the TV industry in the last decade. It now provides a pointer to where the rest of the industry may move next as it looks to develop a converged backend for TV services that are currently running over multiple access networks, with the possibility that eventually all video services might be delivered as HTTP.

Luc Barnaud, VP, TV at Orange Technocentre, says the key differentiator for Orange in the TV market today is the way it can provide a complete TV experience. This starts by combining live and on-demand, with VOD, SVOD and catch-up TV still proving a strong consumer benefit compared to broadcast-centric network services, he notes. It also means offering services on the TV, PC, mobile, tablets and connected TVs. Finally, Orange makes its services accessible on the portable devices outside the home as well as inside. “Allowing customers to use their service on multiple screens in different places represents a compelling option,” Barnaud declares.

With the growing popularity of over-the-top video and the increasing penetration of connected TV devices, all service providers now need a strategy to both counter and embrace online video, and potentially monetize it. Orange is no exception. The company provides a content portal on LG televisions in France already, something that gives existing Orange customers and other consumers more opportunities to engage with the brand. It is also engaged with other device manufacturers.

Meanwhile, like a number of major telcos, the company will use its own CDN (Content Delivery Network) to ensure it has a stake in the delivery of OTT video (and the opportunity to improve the Quality of Experience for online services). “CDN infrastructure is a natural extension of our networks and service platforms and leverages our historical skills as an operator,” Barnaud says, pointing to QoS management, 24/24 availability and systems reliability among other core Orange competencies.

The company is also gearing itself to integrate Web-powered entertainment experiences into the IPTV offering more closely. Barnaud explains: “We have had a browser-based strategy for several years and can use our expertise to provide an enriched TV experience that can take advantage of content from the Web, like interactive TV ads connecting to third-party Web servers, and specific events that mix live and Web content.” He says Orange is currently working on a content recommendation solution that harnesses social networks, providing an example of how it can go further in exploiting Web content.

“Our approach to integrating Web services on TV remains very TV-centric,” he adds. “We aim to deliver relevant and tailored services to the customer. Today, the Web as we know it on PC is not easy to use on a TV. Our job is to select the right services and make them relevant on TV in terms of usage.”

Barnaud points out that overall, it is the guarantee of a premium quality service that sets Orange apart from today’s over-the-top services, however. He highlights the company’s end-to-end control of the content distribution chain and the associated services, from the content platform and portals to the end user device, including network elements covering CDNs, routers, DSLAMs and domestic gateways. In short, the company has the infrastructure to compete on every level, whether via broadband or the managed network.

One of the next big challenges for Orange, and one that will confront most major Pay TV operators in the next few years, is how to rationalise the service delivery that is now encompassing multiple access networks, managed and unmanaged video and an increasing array of devices. Barnaud reveals: “We are working on the convergence of our content platforms, starting with the unification of the content management and other service enablers. We are also working on the convergence of the content specific protocols that were previously different according to the screen and access used.”

With Orange’s TV service now available via FTTH and DSL, and live feeds coming over satellite and digital terrestrial as well as IPTV, the company is also migrating to a unique core platform for all its fixed TV services. Barnaud acknowledges the growing interest in the industry around the possibility that all video services across all platforms could converge around HTTP adaptive streaming, including for set-top boxes. He says: “Given the success of the protocols and formats that emerged from the web and IT actors, it is difficult not to consider that kind of convergence.”

He says the challenge, as a network and service operator, is to find the balance that delivers the benefits of open protocols and softwares and the performance, security and quality that is enabled by a good vertical, end-to-end integration of content, platforms, networks and devices.

By John Moulding, Videonet