ginger’s thoughts

Les articles publiés sur le site

• HTML5 multi-track audio or video

  2 juin 2012, par silvia
  In the last months, we’ve been working hard at the WHATWG and W3C to spec out new HTML markup and a JavaScript interface for dealing with audio or video content that has more than just one audio and video track. This is particularly relevant when a Web page author wants to add a sign language (...)

• A systematic approach to making Web Applications accessible

  22 février 2012, par silvia

  With the latest developments in HTML5 and the still fairly new ARIA (Accessible Rich Interface Applications) attributes introduced by the W3C WAI (Web Accessibility Initiative), browsers have now implemented many features that allow you to make your JavaScript-heavy Web applications accessible.

  Since I began working on making a complex web application accessible just over a year ago, I discovered that there was no step-by-step guide to approaching the changes necessary for creating an accessible Web application. Therefore, many people believe that it is still hard, if not impossible, to make Web applications accessible. In fact, it can be approached systematically, as this article will describe.

  This post is based on a talk that Alice Boxhall and I gave at the recent Linux.conf.au titled “Developing accessible Web apps – how hard can it be?” (slides, video), which in turn was based on a Google Developer Day talk by Rachel Shearer (slides).

  These talks, and this article, introduce a process that you can follow to make your Web applications accessible: each step will take you closer to having an application that can be accessed using a keyboard alone, and by users of screenreaders and other accessibility technology (AT).

  The recommendations here only roughly conform to the requirements of WCAG (Web Content Accessibility Guidelines), which is the basis of legal accessibility requirements in many jurisdictions. The steps in this article may or may not be sufficient to meet a legal requirement. It is focused on the practical outcome of ensuring users with disabilities can use your Web application.

  Step-by-step Approach

  The steps to follow to make your Web apps accessible are as follows:

  1. Use native HTML tags wherever possible
  2. Make interactive elements keyboard accessible
  3. Provide extra markup for AT (accessibility technology)

  If you are a total newcomer to accessibility, I highly recommend installing a screenreader and just trying to read/navigate some Web pages. On Windows you can install the free NVDA screenreader, on Mac you can activate the pre-installed VoiceOver screenreader, on Linux you can use Orca, and if you just want a browser plugin for Chrome try installing ChromeVox.

  1. Use native HTML tags

  As you implement your Web application with interactive controls, try to use as many native HTML tags as possible.

  HTML5 provides a rich set of elements which can be used to both add functionality and provide semantic context to your page. HTML4 already included many useful interactive controls, like <a>, <button>, <input> and <select>, and semantic landmark elements like <h1>. HTML5 adds richer <input> controls, and a more sophisticated set of semantic markup elements like such as <time>, <progress>, <meter>, <nav>, <header>, <article> and <aside>. (Note: check browser support for browser support of the new tags).

  Using as much of the rich HTML5 markup as possible means that you get all of the accessibility features which have been implemented in the browser for those elements, such as keyboard support, short-cut keys and accessibility metadata, for free. For generic tags you have to implement them completely from scratch.

  What exactly do you miss out on when you use a generic tag such as <div> over a specific semantic one such as <button>?

  1. Generic tags are not focusable. That means you cannot reach them through using the [tab] on the keyboard.
  2. You cannot activate them with the space bar or enter key or perform any other keyboard interaction that would be regarded as typical with such a control.
  3. Since the role that the control represents is not specified in code but is only exposed through your custom visual styling, screenreaders cannot express to their users what type of control it is, e.g. button or link.
  4. Neither can screenreaders add the control to the list of controls on the page that are of a certain type, e.g. to navigate to all headers of a certain level on the page.
  5. And finally you need to manually style the element in order for it to look distinctive compared to other elements on the page; using a default control will allow the browser to provide the default style for the platform, which you can still override using CSS if you want.

  Example:

  Compare these two buttons. The first one is implemented using a <div> tag, the second one using a <button> tag. Try using a screenreader to experience the difference.

  Send	<style> .custombutton { cursor: pointer; border: 1px solid #000; background-color: #F6F6F6; display: inline-block; padding: 2px 5px; } </style> <div class="custombutton" onclick="alert('sent!')"> Send </div>
  Send	<button onclick="alert('sent!')"> Send </button>

  2. Make interactive elements keyboard accessible

  Many sophisticated web applications have some interactive controls that just have no appropriate HTML tag equivalent. In this case, you will have had to build an interactive element with JavaScript and <div> and/or <span> tags and lots of custom styling. The good news is, it’s possible to make even these custom controls accessible, and as a side benefit you will also make your application smoother to use for power users.

  The first thing you can do to test usability of your control, or your Web app, is to unplug the mouse and try to use only the [TAB] and [ENTER] keys to interact with your application.

  

  Try the following:

  • Can you reach all interactive elements with [TAB]?
  • Can you activate interactive elements with [ENTER] (or [SPACE])?
  • Are the elements in the right tab order?
  • After interaction: is the right element in focus?
  • Is there a keyboard shortcut that activates the element (accesskey)?

  No? Let’s fix it.

  2.1. Reaching interactive elements

  If you have an element on your page that cannot be reached with [TAB], put a @tabindex attribute on it.

  Example:

  Here we have a <span> tag that works as a link (don’t do this – it’s just a simple example). The first one cannot be reached using [TAB] but the second one has a tabindex and is thus part of the tab order of the HTML page.

  (Note: since we experiment lots with the tabindex in this article, to avoid confusion, click on some text in this paragraph and then hit the [TAB] key to see where it goes next. The click will set your keyboard focus in the DOM.)

  Click	<style> .customlink { text-decoration: underline; cursor: pointer; } </style> <span class="customlink" onclick="alert('activated!')"> Click </span>
  Click	<style> .customlink { text-decoration: underline; cursor: pointer; } </style> <span class="customlink" onclick="alert('activated!')" tabindex="0"> Click </span>

  You set @tabindex=0 to add an element into the native tab order of the page, which is the DOM order.

  2.2. Activating interactive elements

  Next, you typically want to be able to use the [ENTER] and [SPACE] keys to activate your custom control. To do so, you will need to implement an onkeydown event handler. Note that the keyCode for [ENTER] is 13 and for [SPACE] is 32.

  Example:

  Let’s add this functionality to the <span> tag from before. Try tabbing to it and hit the [ENTER] or [SPACE] key.

  Click	<span class="customlink" onclick="alert('activated!')" tabindex="0"> Click </span>
  <script> function handlekey(event) { var target = event.target || event.srcElement; if (event.keyCode == 13 || event.keyCode == 32) { target.onclick(); } } </script> Click	<span class="customlink" onclick="alert('activated!')" tabindex="0" onkeydown="handlekey(event);"> Click </span> <script> function handlekey(event) { var target = event.target || event.srcElement; if (event.keyCode == 13 || event.keyCode == 32) { target.onclick(); } } </script>

  Note that there are some controls that might need support for keys other than [tab] or [enter] to be able to use them from the keyboard alone, for example a custom list box, menu or slider should respond to arrow keys.

  2.3. Elements in the right tab order

  Have you tried tabbing to all the elements on your page that you care about? If so, check if the order of tab stops seems right. The default order is given by the order in which interactive elements appear in the DOM. For example, if your page’s code has a right column that is coded before the main article, then the links in the right column will receive tab focus first before the links in the main article.

  You could change this by re-ordering your DOM, but oftentimes this is not possible. So, instead give the elements that should be the first ones to receive tab focus a positive @tabindex. The tab access will start at the smallest non-zero @tabindex value. If multiple elements share the same @tabindex value, these controls receive tab focus in DOM order. After that, interactive elements and those with @tabindex=0 will receive tab focus in DOM order.

  Example:

  The one thing that always annoys me the most is if the tab order in forms that I am supposed to fill in is illogical. Here is an example where the first and last name are separated by the address because they are in a table. We could fix it by moving to a <div> based layout, but let’s use @tabindex to demonstrate the change.

  Firstname: Address: Lastname: City:	<table class="customtabs"> <tr> <td>Firstname: <input type="text" id="firstname"> </td> <td>Address: <input type="text" id="address"> </td> </tr> <tr> <td>Lastname: <input type="text" id="lastname"> </td> <td>City: <input type="text" id="city"> </td> </tr> </table>
  Click here to test this form, then [TAB]: Firstname: Address: Lastname: City:	<table class="customtabs"> <tr> <td>Firstname: <input type="text" id="firstname" tabindex="10"> </td> <td>Address: <input type="text" id="address" tabindex="30"> </td> </tr> <tr> <td>Lastname: <input type="text" id="lastname" tabindex="20"> </td> <td>City: <input type="text" id="city" tabindex="40"> </td> </tr> </table>

  Be very careful with using non-zero tabindex values. Since they change the tab order on the page, you may get side effects that you might not have intended, such as having to give other elements on the page a non-zero tabindex value to avoid skipping too many other elements as I would need to do here.

  2.4. Focus on the right element

  Some of the controls that you create may be rather complex and open elements on the page that were previously hidden. This is particularly the case for drop-downs, pop-ups, and menus in general. Oftentimes the hidden element is not defined in the DOM right after the interactive control, such that a [TAB] will not put your keyboard focus on the next element that you are interacting with.

  The solution is to manage your keyboard focus from JavaScript using the .focus() method.

  Example:

  Here is a menu that is declared ahead of the menu button. If you tab onto the button and hit enter, the menu is revealed. But your tab focus is still on the menu button, so your next [TAB] will take you somewhere else. We fix it by setting the focus on the first menu item after opening the menu.

  Menu item1 Menu item2 Menu <script> function displayMenu(value) { document.getElementById("custommenu").style.display=value; } </script>	<div id="custommenu" style="display:none;"> <button id="item1" onclick="displayMenu('none');">Menu item1</button> <button id="item2" onclick="displayMenu('none');">Menu item2</button> </div> <button onclick="displayMenu('block');">Menu</button> <script> function displayMenu(value) { document.getElementById("custommenu").style.display=value; } </script>
  Menu item1 Menu item2 Menu <script> function displayMenu2(value) { document.getElementById("custommenu2").style.display=value; document.getElementById("item1").focus(); } </script>	<div id="custommenu" style="display:none;"> <button id="item1" onclick="displayMenu('none');">Menu item1</button> <button id="item2" onclick="displayMenu('none');">Menu item2</button> </div> <button onclick="displayMenu('block');">Menu</button> <script> function displayMenu(value) { document.getElementById("custommenu").style.display=value; document.getElementById("item1").focus(); } </script>

  You will notice that there are still some things you can improve on here. For example, after you close the menu again with one of the menu items, the focus does not move back onto the menu button.

  Also, after opening the menu, you may prefer not to move the focus onto the first menu item but rather just onto the menu <div>. You can do so by giving that div a @tabindex and then calling .focus() on it. If you do not want to make the div part of the normal tabbing order, just give it a @tabindex=-1 value. This will allow your div to receive focus from script, but be exempt from accidental tabbing onto (though usually you just want to use @tabindex=0).

  Bonus: If you want to help keyboard users even more, you can also put outlines on the element that is currently in focus using CSS”s outline property. If you want to avoid the outlines for mouse users, you can dynamically add a class that removes the outline in mouseover events but leaves it for :focus.

  2.5. Provide sensible keyboard shortcuts

  At this stage your application is actually keyboard accessible. Congratulations!

  However, it’s still not very efficient: like power-users, screenreader users love keyboard shortcuts: can you imagine if you were forced to tab through an entire page, or navigate back to a menu tree at the top of the page, to reach each control you were interested in? And, obviously, anything which makes navigating the app via the keyboard more efficient for screenreader users will benefit all power users as well, like the ubiquitous keyboard shortcuts for cut, copy and paste.

  HTML4 introduced so-called accesskeys for this. In HTML5 @accesskey is now allowed on all elements.

  The @accesskey attribute takes the value of a keyboard key (e.g. @accesskey="x") and is activated through platform- and browser-specific activation keys. For example, on the Mac it’s generally the [Ctrl] key, in IE it’ the [Alt] key, in Firefox on Windows [Shift]-[Alt], and in Opera on Windows [Shift]-[ESC]. You press the activation key and the accesskey together which either activates or focuses the element with the @accesskey attribute.

  Example:

  Send <script> var button = document.getElementById('accessbutton'); if (button.accessKeyLabel) { button.innerHTML += ' (' + button.accessKeyLabel + ')'; } </script>

  <button id="accessbutton" onclick="alert('sent!')" accesskey="e"> Send </button> <script> var button = document.getElementById('accessbutton'); if (button.accessKeyLabel) { button.innerHTML += ' (' + button.accessKeyLabel + ')'; } </script>

  Now, the idea behind this is clever, but the execution is pretty poor. Firstly, the different activation keys between different platforms and browsers make it really hard for people to get used to the accesskeys. Secondly, the key combinations can conflict with browser and screenreader shortcut keys, the first of which will render browser shortcuts unusable and the second will effectively remove the accesskeys.

  In the end it is up to the Web application developer whether to use the accesskey attribute or whether to implement explicit shortcut keys for the application through key event handlers on the window object. In either case, make sure to provide a help list for your shortcut keys.

  Also note that a page with a really good hierarchical heading layout and use of ARIA landmarks can help to eliminate the need for accesskeys to jump around the page, since there are typically default navigations available in screen readers to jump directly to headings, hyperlinks, and ARIA landmarks.

  3. Provide markup for AT

  Having made the application keyboard accessible also has advantages for screenreaders, since they can now reach the controls individually and activate them. So, next we will use a screenreader and close our eyes to find out where we only provide visual cues to understand the necessary interaction.

  Here are some of the issues to consider:

  • Role may need to get identified
  • States may need to be kept track of
  • Properties may need to be made explicit
  • Labels may need to be provided for elements

  This is where the W3C’s ARIA (Accessible Rich Internet Applications) standard comes in. ARIA attributes provide semantic information to screen readers and other AT that is otherwise conveyed only visually.

  Note that using ARIA does not automatically implement the standard widget behavior – you’ll still need to add focus management, keyboard navigation, and change aria attribute values in script.

  3.1. ARIA roles

  After implementing a custom interactive widget, you need to add a @role attribute to indicate what type of controls it is, e.g. that it is playing the role of a standard tag such as a button.

  Example:

  This menu button is implemented as a <div>, but with a role of “button” it is announced as a button by a screenreader.

  Menu	<div tabindex="0" role="button">Menu</div>

  ARIA roles also describe composite controls that do not have a native HTML equivalent.

  Example:

  This menu with menu items is implemented as a set of <div> tags, but with a role of “menu” and “menuitem” items.

  Cut Copy Paste

  <div role="menu"> <div tabindex="0" role="menuitem">Cut</div> <div tabindex="0" role="menuitem">Copy</div> <div tabindex="0" role="menuitem">Paste</div> </div>

  3.2. ARIA states

  Some interactive controls represent different states, e.g. a checkbox can be checked or unchecked, or a menu can be expanded or collapsed.

  Example:

  The following menu has states on the menu items, which are here not just used to give an aural indication through the screenreader, but also a visual one through CSS.

  Left Center Right

  <style> .custombutton[aria-checked=true]:before { content: "\2713 "; } </style> <div role="menu"> <div tabindex="0" role="menuitem" aria-checked="true">Left</div> <div tabindex="0" role="menuitem" aria-checked="false">Center</div> <div tabindex="0" role="menuitem" aria-checked="false">Right</div> </div>

  3.3. ARIA properties

  Some of the functionality of interactive controls cannot be captured by the role attribute alone. We have ARIA properties to add features that the screenreader needs to announce, such as aria-label, aria-haspopup, aria-activedescendant, or aria-live.

  Example:

  The following drop-down menu uses aria-haspopup to tell the screenreader that there is a popup hidden behind the menu button together with an ARIA state of aria-expanded to track whether it’s open or closed.

  Justify Left Center Right <script> var button = document.getElementById("button"); var menu = document.getElementById("menu"); var items = document.getElementsByClassName("menuitem"); var focused = 0; function showMenu(evt) { evt.stopPropagation(); menu.style.visibility = 'visible'; button.setAttribute('aria-expanded','true'); focused = getSelected(); items[focused].focus(); } function hideMenu(evt) { evt.stopPropagation(); menu.style.visibility = 'hidden'; button.setAttribute('aria-expanded','false'); button.focus(); } function getSelected() { for (var i=0; i < items.length; i++) { if (items[i].getAttribute('aria-checked') == 'true') { return i; } } } function setSelected(elem) { var curSelected = getSelected(); items[curSelected].setAttribute('aria-checked', 'false'); elem.setAttribute('aria-checked', 'true'); } function selectItem(evt) { setSelected(evt.target); hideMenu(evt); } function getPrevItem(index) { var prev = index - 1; if (prev < 0) { prev = items.length - 1; } return prev; } function getNextItem(index) { var next = index + 1; if (next == items.length) { next = 0; } return next; } function handleButtonKeys(evt) { evt.stopPropagation(); var key = evt.keyCode; switch(key) { case (13): /* ENTER */ case (32): /* SPACE */ showMenu(evt); default: } } function handleMenuKeys(evt) { evt.stopPropagation(); var key = evt.keyCode; switch(key) { case (38): /* UP */ focused = getPrevItem(focused); items[focused].focus(); break; case (40): /* DOWN */ focused = getNextItem(focused); items[focused].focus(); break; case (13): /* ENTER */ case (32): /* SPACE */ setSelected(evt.target); hideMenu(evt); break; case (27): /* ESC */ hideMenu(evt); break; default: } } button.addEventListener('click', showMenu, false); button.addEventListener('keydown', handleButtonKeys, false); for (var i = 0; i < items.length; i++) { items[i].addEventListener('click', selectItem, false); items[i].addEventListener('keydown', handleMenuKeys, false); } </script>

  <div class="custombutton" id="button" tabindex="0" role="button" aria-expanded="false" aria-haspopup="true"> <span>Justify</span> </div> <div role="menu" class="menu" id="menu" style="display: none;"> <div tabindex="0" role="menuitem" class="menuitem" aria-checked="true"> Left </div> <div tabindex="0" role="menuitem" class="menuitem" aria-checked="false"> Center </div> <div tabindex="0" role="menuitem" class="menuitem" aria-checked="false"> Right </div> </div> [CSS and JavaScript for example omitted]

  3.4. Labelling

  The main issue that people know about accessibility seems to be that they have to put alt text onto images. This is only one means to provide labels to screenreaders for page content. Labels are short informative pieces of text that provide a name to a control.

  There are actually several ways of providing labels for controls:

  • on img elements use @alt
  • on input elements use the label element
  • use @aria-labelledby if there is another element that contains the label
  • use @title if you also want a label to be used as a tooltip
  • otherwise use @aria-label

  I'll provide examples for the first two use cases - the other use cases are simple to deduce.

  Example:

  The following two images show the rough concept for providing alt text for images: images that provide information should be transcribed, images that are just decorative should receive an empty @alt attribute.

  Image by Noah Sussman

  <img src="texture.jpg" alt=""> <img src="lolcat.jpg" alt="shocked lolcat titled 'HTML cannot do that!"> <img src="texture.jpg" alt="">

  When marking up decorative images with an empty @alt attribute, the image is actually completely removed from the accessibility tree and does not confuse the blind user. This is a desired effect, so do remember to mark up all your images with @alt attributes, even those that don't contain anything of interest to AT.

  Example:

  In the example form above in Section 2.3, when tabbing directly on the input elements, the screen reader will only say "edit text" without announcing what meaning that text has. That's not very useful. So let's introduce a label element for the input elements. We'll also add checkboxes with a label.

  Doctor title: Firstname: Lastname: Address: City: Remember me:

  <label>Doctor title:</label> <input type="checkbox" id="doctor"/> <label>Firstname:</label> <input type="text" id="firstname2"/> <label for="lastname2">Lastname:</label> <input type="text" id="lastname2"/> <label>Address: <input type="text" id="address2"> </label> <label for="city2">City: <input type="text" id="city2"> </label> <label for="remember">Remember me:</label> <input type="checkbox" id="remember">

  In this example we use several different approaches to show what a different it makes to use the <label> element to mark up input boxes.

  The first two fields just have a <label> element next to a <input> element. When using a screenreader you will not notice a difference between this and not using the <label> element because there is no connection between the <label> and the <input> element.

  In the third field we use the @for attribute to create that link. Now the input field isn't just announced as "edit text", but rather as "Lastname edit text", which is much more useful. Also, the screenreader can now skip the labels and get straight on the input element.

  In the fourth and fifth field we actually encapsulate the <input> element inside the <label> element, thus avoiding the need for a @for attribute, though it doesn't hurt to explicity add it.

  Finally we look at the checkbox. By including a referenced <label> element with the checkbox, we change the screenreaders announcement from just "checkbox not checked" to "Remember me checkbox not checked". Also notice that the click target now includes the label, making the checkbox not only more usable to screenreaders, but also for mouse users.

  4. Conclusions

  This article introduced a process that you can follow to make your Web applications accessible. As you do that, you will noticed that there are other things that you may need to do in order to give the best experience to a power user on a keyboard, a blind user using a screenreader, or a vision-impaired user using a screen magnifier. But once you've made a start, you will notice that it's not all black magic and a lot can be achieved with just a little markup.

  You will find more markup in the WAI ARIA specification and many more resources at Mozilla's ARIA portal. Now go and change the world!

  Many thanks to Alice Boxhall and Dominic Mazzoni for their proof-reading and suggested changes that really helped improve the article!

• Protected : A systematic approach to making Web Applications accessible

  13 février 2012, par silvia
  There is no excerpt because this is a protected post.

• My crazy linux.conf.au week

  9 février 2012, par silvia

  In January I attended the annual Australian Linux and Open Source conference (LCA). But since I was sick all of January and had a lot to catch up on, I never got around to sharing all the talks that I gave during that time.

  Drupal Down Under

  It started with a talk at Drupal Down Under, which happened the weekend before LCA. I gave a talk titled “HTML5 video specifications” (video, slides).

  I spoke about the video and audio element in HTML5, how to provide fallback content, how to encode content, how to control them from JavaScript, and briefly about Drupal video modules, though the next presentation provided much more insight into those. I explained how to make the HTML5 media elements accessible, including accessible controls, captions, audio descriptions, and the new WebVTT file format. I ran out of time to introduce the last section of my slides which are on WebRTC.

  Linux.conf.au

  On the first day of LCA I gave a talk both in the Multimedia Miniconf and the Browser Miniconf.

  Browser Miniconf

  In the Browser Miniconf I talked about “Web Standardisation – how browser vendors collaborate, or not” (slides). Maybe the most interesting part about this was that I tried out a new slide “deck” tool called impress.js. I’m not yet sure if I like it but it worked well for this talk, in which I explained how the HTML5 spec is authored and who has input.

  I also sat on a panel of browser developers in the Browser Miniconf (more as a standards than as a browser developer, but that’s close enough). We were asked about all kinds of latest developments in HTML5, CSS3, and media standards in the browser.

  Multimedia Miniconf

  In the Multimedia Miniconf I gave a “HTML5 media accessibility update” (slides). I talked about the accessibility problems of Flash, how native HTML5 video players will be better, about accessible video controls, captions, navigation chapters, audio descriptions, and WebVTT. I also provided a demo of how to synchronize multiple video elements using a polyfill for the multitrack API.

  I also provided an update on HTTP adaptive streaming APIs as a lightning talk in the Multimedia Miniconf. I used an extract of the Drupal conference slides for it.

  Main conference

  Finally, and most importantly, Alice Boxhall and myself gave a talk in the main linux.conf.au titled “Developing Accessible Web Apps – how hard can it be?” (video, slides). I spoke about a process that you can follow to make your Web applications accessible. I’m writing a separate blog post to explain this in more detail. In her part, Alice dug below the surface of browsers to explain how the accessibility markup that Web developers provide is transformed into data structures that are handed to accessibility technologies.

• Open Media Developers Track at OVC 2011

  11 octobre 2011, par silvia

  The Open Video Conference that took place on 10-12 September was so overwhelming, I’ve still not been able to catch my breath! It was a dense three days for me, even though I only focused on the technology sessions of the conference and utterly missed out on all the policy and content discussions.

  Roughly 60 people participated in the Open Media Software (OMS) developers track. This was an amazing group of people capable and willing to shape the future of video technology on the Web:

  • HTML5 video developers from Apple, Google, Opera, and Mozilla (though we missed the NZ folks),
  • codec developers from WebM, Xiph, and MPEG,
  • Web video developers from YouTube, JWPlayer, Kaltura, VideoJS, PopcornJS, etc.,
  • content publishers from Wikipedia, Internet Archive, YouTube, Netflix, etc.,
  • open source tool developers from FFmpeg, gstreamer, flumotion, VideoLAN, PiTiVi, etc,
  • and many more.

  To provide a summary of all the discussions would be impossible, so I just want to share the key take-aways that I had from the main sessions.

  WebRTC: Realtime Communications and HTML5

  Tim Terriberry (Mozilla), Serge Lachapelle (Google) and Ethan Hugg (CISCO) moderated this session together (slides). There are activities both at the W3C and at IETF – the ones at IETF are supposed to focus on protocols, while the W3C ones on HTML5 extensions.

  The current proposal of a PeerConnection API has been implemented in WebKit/Chrome as open source. It is expected that Firefox will have an add-on by Q1 next year. It enables video conferencing, including media capture, media encoding, signal processing (echo cancellation etc), secure transmission, and a data stream exchange.

  Current discussions are around the signalling protocol and whether SIP needs to be required by the standard. Further, the codec question is under discussion with a question whether to mandate VP8 and Opus, since transcoding gateways are not desirable. Another question is how to measure the quality of the connection and how to report errors so as to allow adaptation.

  What always amazes me around RTC is the sheer number of specialised protocols that seem to be required to implement this. WebRTC does not disappoint: in fact, the question was asked whether there could be a lighter alternative than to re-use dozens of years of protocol development – is it over-engineered? Can desktop players connect to a WebRTC session?

  We are already in a second or third revision of this part of the HTML5 specification and yet it seems the requirements are still being collected. I’m quietly confident that everything is done to make the lives of the Web developer easier, but it sure looks like a huge task.

  The Missing Link: Flash to HTML5

  Zohar Babin (Kaltura) and myself moderated this session and I must admit that this session was the biggest eye-opener for me amongst all the sessions. There was a large number of Flash developers present in the room and that was great, because sometimes we just don’t listen enough to lessons learnt in the past.

  This session gave me one of those aha-moments: it the form of the Flash appendBytes() API function.

  The appendBytes() function allows a Flash developer to take a byteArray out of a connected video resource and do something with it – such as feed it to a video for display. When I heard that Web developers want that functionality for JavaScript and the video element, too, I instinctively rejected the idea wondering why on earth would a Web developer want to touch encoded video bytes – why not leave that to the browser.

  But as it turns out, this is actually a really powerful enabler of functionality. For example, you can use it to:

  • display mid-roll video ads as part of the same video element,
  • sequence playlists of videos into the same video element,
  • implement DVR functionality (high-speed seeking),
  • do mash-ups,
  • do video editing,
  • adaptive streaming.

  This totally blew my mind and I am now completely supportive of having such a function in HTML5. Together with media fragment URIs you could even leave all the header download management for resources to the Web browser and just request time ranges from a video through an appendBytes() function. This would be easier on the Web developer than having to deal with byte ranges and making sure that appropriate decoding pipelines are set up.

  Standards for Video Accessibility

  Philip Jagenstedt (Opera) and myself moderated this session. We focused on the HTML5 track element and the WebVTT file format. Many issues were identified that will still require work.

  One particular topic was to find a standard means of rendering the UI for caption, subtitle, und description selection. For example, what icons should be used to indicate that subtitles or captions are available. While this is not part of the HTML5 specification, it’s still important to get this right across browsers since otherwise users will get confused with diverging interfaces.

  Chaptering was discussed and a particular need to allow URLs to directly point at chapters was expressed. I suggested the use of named Media Fragment URLs.

  The use of WebVTT for descriptions for the blind was also discussed. A suggestion was made to use the voice tag <v> to allow for “styling” (i.e. selection) of the screen reader voice.

  Finally, multitrack audio or video resources were also discussed and the @mediagroup attribute was explained. A question about how to identify the language used in different alternative dubs was asked. This is an issue because @srclang is not on audio or video, only on text, so it’s a missing feature for the multitrack API.

  Beyond this session, there was also a breakout session on WebVTT and the track element. As a consequence, a number of bugs were registered in the W3C bug tracker.

  WebM: Testing, Metrics and New features

  This session was moderated by John Luther and John Koleszar, both of the WebM Project. They started off with a presentation on current work on WebM, which includes quality testing and improvements, and encoder speed improvement. Then they moved on to questions about how to involve the community more.

  The community criticised that communication of what is happening around WebM is very scarce. More sharing of information was requested, including a move to using open Google+ hangouts instead of Google internal video conferences. More use of the public bug tracker can also help include the community better.

  Another pain point of the community was that code is introduced and removed without much feedback. It was requested to introduce a peer review process. Also it was requested that example code snippets are published when new features are announced so others can replicate the claims.

  This all indicates to me that the WebM project is increasingly more open, but that there is still a lot to learn.

  Standards for HTTP Adaptive Streaming

  This session was moderated by Frank Galligan and Aaron Colwell (Google), and Mark Watson (Netflix).

  Mark started off by giving us an introduction to MPEG DASH, the MPEG file format for HTTP adaptive streaming. MPEG has just finalized the format and he was able to show us some examples. DASH is XML-based and thus rather verbose. It is covering all eventualities of what parameters could be switched during transmissions, which makes it very broad. These include trick modes e.g. for fast forwarding, 3D, multi-view and multitrack content.

  MPEG have defined profiles – one for live streaming which requires chunking of the files on the server, and one for on-demand which requires keyframe alignment of the files. There are clear specifications for how to do these with MPEG. Such profiles would need to be created for WebM and Ogg Theora, too, to make DASH universally applicable.

  Further, the Web case needs a more restrictive adaptation approach, since the video element’s API is already accounting for some of the features that DASH provides for desktop applications. So, a Web-specific profile of DASH would be required.

  Then Aaron introduced us to the MediaSource API and in particular the webkitSourceAppend() extension that he has been experimenting with. It is essentially an implementation of the appendBytes() function of Flash, which the Web developers had been asking for just a few sessions earlier. This was likely the biggest announcement of OVC, alas a quiet and technically-focused one.

  Aaron explained that he had been trying to find a way to implement HTTP adaptive streaming into WebKit in a way in which it could be standardised. While doing so, he also came across other requirements around such chunked video handling, in particular around dynamic ad insertion, live streaming, DVR functionality (fast forward), constraint video editing, and mashups. While trying to sort out all these requirements, it became clear that it would be very difficult to implement strategies for stream switching, buffering and delivery of video chunks into the browser when so many different and likely contradictory requirements exist. Also, once an approach is implemented and specified for the browser, it becomes very difficult to innovate on it.

  Instead, the easiest way to solve it right now and learn about what would be necessary to implement into the browser would be to actually allow Web developers to queue up a chunk of encoded video into a video element for decoding and display. Thus, the webkitSourceAppend() function was born (specification).

  The proposed extension to the HTMLMediaElement is as follows:

  partial interface HTMLMediaElement {
    // URL passed to src attribute to enable the media source logic.
    readonly attribute [URL] DOMString webkitMediaSourceURL;
  
    bool webkitSourceAppend(in Uint8Array data);
  
    // end of stream status codes.
    const unsigned short EOS_NO_ERROR = 0;
    const unsigned short EOS_NETWORK_ERR = 1;
    const unsigned short EOS_DECODE_ERR = 2;
  
    void webkitSourceEndOfStream(in unsigned short status);
  
    // states
    const unsigned short SOURCE_CLOSED = 0;
    const unsigned short SOURCE_OPEN = 1;
    const unsigned short SOURCE_ENDED = 2;
  
    readonly attribute unsigned short webkitSourceState;
  };

  The code is already checked into WebKit, but commented out behind a command-line compiler flag.

  Frank then stepped forward to show how webkitSourceAppend() can be used to implement HTTP adaptive streaming. His example uses WebM – there are no examples with MPEG or Ogg yet.

  The chunks that Frank’s demo used were 150 video frames long (6.25s) and 5s long audio. Stream switching only switched video, since audio data is much lower bandwidth and more important to retain at high quality. Switching was done on multiplexed files.

  Every chunk requires an XHR range request – this could be optimised if the connections were kept open per adaptation. Seeking works, too, but since decoding requires download of a whole chunk, seeking latency is determined by the time it takes to download and decode that chunk.

  Similar to DASH, when using this approach for live streaming, the server has to produce one file per chunk, since byte range requests are not possible on a continuously growing file.

  Frank did not use DASH as the manifest format for his HTTP adaptive streaming demo, but instead used a hacked-up custom XML format. It would be possible to use JSON or any other format, too.

  After this session, I was actually completely blown away by the possibilities that such a simple API extension allows. If I wasn’t sold on the idea of a appendBytes() function in the earlier session, this one completely changed my mind. While I still believe we need to standardise a HTTP adaptive streaming file format that all browsers will support for all codecs, and I still believe that a native implementation for support of such a file format is necessary, I also believe that this approach of webkitSourceAppend() is what HTML needs – and maybe it needs it faster than native HTTP adaptive streaming support.

  Standards for Browser Video Playback Metrics

  This session was moderated by Zachary Ozer and Pablo Schklowsky (JWPlayer). Their motivation for the topic was, in fact, also HTTP adaptive streaming. Once you leave the decisions about when to do stream switching to JavaScript (through a function such a wekitSourceAppend()), you have to expose stream metrics to the JS developer so they can make informed decisions. The other use cases is, of course, monitoring of the quality of video delivery for reporting to the provider, who may then decide to change their delivery environment.

  The discussion found that we really care about metrics on three different levels:

  • measuring the network performance (bandwidth)
  • measuring the decoding pipeline performance
  • measuring the display quality

  In the end, it seemed that work previously done by Steve Lacey on a proposal for video metrics was generally acceptable, except for the playbackJitter metric, which may be too aggregate to mean much.

  Device Inputs / A/V in the Browser

  I didn’t actually attend this session held by Anant Narayanan (Mozilla), but from what I heard, the discussion focused on how to manage permission of access to video camera, microphone and screen, e.g. when multiple applications (tabs) want access or when the same site wants access in a different session. This may apply to real-time communication with screen sharing, but also to photo sharing, video upload, or canvas access to devices e.g. for time lapse photography.

  Open Video Editors

  This was another session that I wasn’t able to attend, but I believe the creation of good open source video editing software and similar video creation software is really crucial to giving video a broader user appeal.

  Jeff Fortin (PiTiVi) moderated this session and I was fascinated to later see his analysis of the lifecycle of open source video editors. It is shocking to see how many people/projects have tried to create an open source video editor and how many have stopped their project. It is likely that the creation of a video editor is such a complex challenge that it requires a larger and more committed open source project – single people will just run out of steam too quickly. This may be comparable to the creation of a Web browser (see the size of the Mozilla project) or a text processing system (see the size of the OpenOffice project).

  Jeff also mentioned the need to create open video editor standards around playlist file formats etc. Possibly the Open Video Alliance could help. In any case, something has to be done in this space – maybe this would be a good topic to focus next year’s OVC on?

  Monday’s Breakout Groups

  The conference ended officially on Sunday night, but we had a third day of discussions / hackday at the wonderful New York Lawschool venue. We had collected issues of interest during the two previous days and organised the breakout groups on the morning (Schedule).

  In the Content Protection/DRM session, Mark Watson from Netflix explained how their API works and that they believe that all we need in browsers is a secure way to exchange keys and an indicator of protection scheme is used – the actual protection scheme would not be implemented by the browser, but be provided by the underlying system (media framework/operating system). I think that until somebody actually implements something in a browser fork and shows how this can be done, we won’t have much progress. In my understanding, we may also need to disable part of the video API for encrypted content, because otherwise you can always e.g. grab frames from the video element into canvas and save them from there.

  In the Playlists and Gapless Playback session, there was massive brainstorming about what new cool things can be done with the video element in browsers if playback between snippets can be made seamless. Further discussions were about a standard playlist file formats (such as XSPF, MRSS or M3U), media fragment URIs in playlists for mashups, and the need to expose track metadata for HTML5 media elements.

  What more can I say? It was an amazing three days and the complexity of problems that we’re dealing with is a tribute to how far HTML5 and open video has already come and exciting news for the kind of applications that will be possible (both professional and community) once we’ve solved the problems of today. It will be exciting to see what progress we will have made by next year’s conference.

  Thanks go to Google for sponsoring my trip to OVC.

  UPDATE: We actually have a mailing list for open media developers who are interested in these and similar topics – do join at http://lists.annodex.net/cgi-bin/mailman/listinfo/foms.