Interactive

Building Interactive HTML5 Videos

The HTML5 <video> element makes embedding videos into your site as easy as embedding images. And since all major browsers support <video> since 2011, it’s also the most reliable way to get your moving pictures seen by people.

A more recent addition to the HTML5 family is the <track> element. It’s a sub-element of <video>, intended to make the video timeline more accessible. Its main use case is adding closed captions. These captions are loaded from a separate text file (a WebVTT file) and printed over the bottom of the video display. Ian Devlin has written an excellent article on the subject.

Beyond captions though, the <track> element can be used for any kind of interaction with the video timeline. This article explores 3 examples: chapter markers, preview thumbnails, and a timeline search. By the end, you will have sufficient understanding of the <track> element and its scripting API to build your own interactive video experiences.

Chapter Markers

Let’s start with an example made popular by DVD disks: chapter markers. These allow viewers to quickly jump to a specific section. It’s especially useful for longer movies like Sintel:

The chapter markers in this example reside in an external VTT file and are loaded on the page through aelement with a kind of **chapters. The track is set to load by default:

<video width="480" height="204" poster="assets/sintel.jpg" controls>
  <source src="assets/sintel.mp4" type="video/mp4">
  <track src="assets/chapters.vtt" kind="chapters" default>
</video>

Next, we use JavaScript to load the cues of the text track, format them, and print them in a controlbar below the video. Note we have to wait until the external VTT file is loaded:

track.addEventListener('load',function() {
    var c = video.textTracks[0].cues;
    for (var i=0; i<c.length; i++) {
      var s = document.createElement("span");
      s.innerHTML = c[i].text;
      s.setAttribute('data-start',c[i].startTime);
      s.addEventListener("click",seek);
      controlbar.appendChild(s);
    }
});

In above code block, we’re adding 2 properties to the list entries to hook up interactivity. First, we set a data attribute to store the start position of the chapter, and second we add a click handler for an external seek function. This function will jump the video to the start position. If the video is not (yet) playing, we’ll make that so:

function seek() {
  video.currentTime = this.getAttribute('data-start');
  if(video.paused){ video.play(); }
};

That’s it! You now have a visual chapter menu for your video, powered by a VTT track. Note the actual live Chapter Markers example has a little bit more logic than described, e.g. to toggle playback of the video on click, to update the controlbar with the video position, and to add some CSS styling.

Preview Thumbnails

This second example shows a cool feature made popular by Hulu and Netflix: preview thumbnails. When mousing over the controlbar (or dragging on mobile), a small preview of the position you’re about to seek to is displayed:

This example is also powered by an external VTT file, loaded in a metadata track. Instead of texts, the cues in this VTT file contain links to a separate JPG image. Each cue could link to a separate image, but in this case we opted to use a single JPG sprite – to keep latency low and management easy. The cues link to the correct section of the sprite by using Media Fragment URIs.Example:

http://example.com/assets/thumbs.jpg?xywh=0,0,160,90

Next, all important logic to get the right thumbnail and display it lives in a mousemove listener for the controlbar:

controlbar.addEventListener('mousemove',function(e) {
  // first we convert from mouse to time position ..
  var p = (e.pageX - controlbar.offsetLeft) * video.duration / 480;
 
  // ..then we find the matching cue..
  var c = video.textTracks[0].cues;
  for (var i=0; i<c.length; i++) {
      if(c[i].startTime <= p && c[i].endTime > p) {
          break;
      };
  }
 
  // ..next we unravel the JPG url and fragment query..
  var url =c[i].text.split('#')[0];
  var xywh = c[i].text.substr(c[i].text.indexOf("=")+1).split(',');
 
  // ..and last we style the thumbnail overlay
  thumbnail.style.backgroundImage = 'url('+c[i].text.split('#')[0]+')';
  thumbnail.style.backgroundPosition = '-'+xywh[0]+'px -'+xywh[1]+'px';
  thumbnail.style.left = e.pageX - xywh[2]/2+'px';
  thumbnail.style.top = controlbar.offsetTop - xywh[3]+8+'px';
  thumbnail.style.width = xywh[2]+'px';
  thumbnail.style.height = xywh[3]+'px';
});

All done! Again, the actual live Preview Thumbnails example contains some additional code. It includes the same logic for toggling playback and seeking, as well as logic to show/hide the thumbnail when mousing in/out of the controlbar.

Timeline Search

Our last example offers yet another way to unlock your content, this time though in-video search:

This example re-uses an existing captions VTT file, which is loaded into a captions track. Below the video and controlbar, we print a basic search form:

<form>
    <input type="search" />
    <button type="submit">Search</button>
</form>

Like with the thumbnails example, all key logic resides in a single function. This time, it’s the event handler for submitting the form:

form.addEventListener('submit',function(e) {
  // First we’ll prevent page reload and grab the cues/query..
  e.preventDefault();
  var c = video.textTracks[0].cues;
  var q = document.querySelector("input").value.toLowerCase();
 
  // ..then we find all matching cues..
  var a = [];
  for(var j=0; j<c.length; j++) {
    if(c[j].text.toLowerCase().indexOf(q) > -1) {
      a.push(c[j]);
    }
  }
 
  // ..and last we highlight matching cues on the controlbar.
  for (var i=0; i<a.length; i++) {
    var s = document.createElement("span");
    s.style.left = (a[i].startTime/video.duration*480-2)+"px";
    bar.appendChild(s);
  }
});

Three time’s a charm! Like with the other ones, the actual live Timeline Search example contains additional code for toggling playback and seeking, as well as a snippet to update the controlbar help text.

Wrapping Up

Above examples should provide you with enough knowledge to build your own interactive videos. For some more inspiration, see our experiments around clickable hot spots, interactive transcripts, or timeline interaction.

Overall, the HTML5 <track> element provides an easy to use, cross-platform way to add interactivity to your videos. And while it definitely takes time to author VTT files and build similar experiences, you will see higher accessibility of and engagement with your videos. Good luck!

View full post on Mozilla Hacks – the Web developer blog

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Inside the Party Bus: Building a Web App with Multiple Live Video Streams + Interactive Graphics

Gearcloud Labs is exploring the use of open technologies to build new kinds of shared video experiences. Party Bus is a demo app that mixes multiple live video streams together with interactive graphics and synchronized audio. We built it using a combination of node.js, WebSockets, WebRTC, WebGL, and Web Audio. This article shares a few things we learned along the way.

User experience

First, take a ride on the Party Bus app to see what it does. You need Firefox or Chrome plus a decent GPU, but if that’s not handy you can get an idea of the app by watching the example video on YouTube.

Since the app uses WebRTC getUserMedia(), you have to give permission for the browser to use your camera. After it starts, the first thing you’ll notice is your own video stream mounted to a seat on the 3D bus (along with live streams from any other concurrent riders). In most scenes, you can manipulate the bus in 3D using the left mouse (change camera angle), scroll wheel (zoom in/out), and right mouse (change camera position). Also try the buttons in the bottom control bar to apply effects to your own video stream: from changing your color, to flipping yourself upside down, bouncing in your seat, etc.

How party bus uses WebRTC

Party Bus uses WebRTC to set up P2P video streams needed for the experience. WebRTC does a great job supporting native video in the browser, and punching out firewalls to enable peer connections (via STUN). But with WebRTC, you also need to provide your own signaler to coordinate which endpoints will participate in a given application session.

The Party Bus app uses a prototype platform we built called Mixology to handle signaling and support the use of dynamic peer topologies. Note that many apps can simply use peer.js, but we are using Mixology to explore new and scalable approaches for combining large numbers of streams in a variety of different connection graphs.

For example, if a rider joins a bus that already has other riders, the system takes care of building the necessary connection paths between the new rider and peers on the same bus, and then notifying all peers through a WebSocket that the new rider needs to be assigned a seat.

Specifically, clients interact with the Mixology signaling server by instantiating a Mixology object

var m = new Mixology(signalerURL);

and then using it to register with the signaler

m.register(['mix-in'], ['mix-out']);

The two arguments give specific input and output stream types supported by the client. Typing inputs and outputs in this way allows Mixology to assemble arbitrary graphs of stream connections, which may vary depending on application requirements. In the case of Party Bus, we’re just using a fully connected mesh among all peers. That is, all clients register with the same input and output types.

The signaler is implemented as a node.js application that maintains a table of registered peers and the connections among them. The signaler can thus take care of handling peer arrivals, departures, and other events — updating other peers as necessary via callback functions. All communications between peers and the signaler are implemented internally using WebSockets, using socket.io.

For example, when a new peer is registered, the server updates the topology table, and uses a callback function to notify other peers that need to know about the new connection.

m.onPeerRegistered = function(peer) { ... }

In this function, peers designated to send streams initiate the WebRTC offer code. Peers designated to receive streams initiate the WebRTC answer code (as well as provide a callback function onAddStream() to be used when the new input stream is ready).

In the case of Party Bus, it’s then up to the app to map the new video stream to the right seat in the 3D bus model, and from then on, apply the necessary 3D transforms using three.js. Similarly, if a rider leaves the bus, the system takes care of notifying other clients that a peer has exited, so they can take appropriate action to remove what would otherwise be a dead video stream in the display.

Party Bus organizes the “riders” on a bus using an array of virtual screens:

var vsArray = new Array(NUM_SEATS);

After registering itself with Mixology, the app receives a callback whenever a new peer video stream becomes available for its bus instance:

function onAddStream(stream, peerId) {
    var i = getNextOpenSeat();
    vsArray[i] = new VScreen(stream, peerId);
}

The Party Bus app creates a virtual screen object for every video stream on the current bus. The incoming streams are associated with DOM video objects in the virtual screen constructor:

function VScreen(stream, id) {
    var v = document.createElement(‘video’);
    v.setAttribute(“id”, “monitor:+ id);
    v.style.visibility = “hidden”;
    v.src = window.URL.createObjectURL(stream);  // binds stream to dom video object
    v.autoplay =true;
    document.body.appendChild(v);
}

Movie or app?

Party Bus uses three.js to draw a double decker bus, along with virtual screens “riding” in the seats. The animation loop runs about two minutes, and consists of about a dozen director “shots”. Throughout the demo, the individual video screens are live, and can be manipulated by each rider. The overall sequence of shots is designed to change scene lighting and present other visual effects, such as bus thrusters which were created with the particle engine of Stemkoski.

Party Bus is a web app, but the animation is programmed so the user can just let it run like a movie. The curious user may try to interact with it, and find that in most scenes it’s also possible to change the 3D view. However, in shots with a moving camera or bus, we found it necessary to block certain camera translations (movements in x, y, z position), or rotations (turning on x, y, z axis) — otherwise, the mouse will “fight” the program, resulting in a jerky presentation.

But most of the fun in Party Bus is just hamming it up for the camera, applying visual effects to your own stream, and looking for other live riders on the bus.

More info

For more information on the Party Bus app, or to stay in the loop on development of the Mixology platform, please check out www.gearcloudlabs.com.

View full post on Mozilla Hacks – the Web developer blog

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Web Professional Trends for 2014 – Web and Interactive Design with Jason Cranford Teague

In this 10 minute interview with Jason Cranford Teague, Author, expert Rosenfeld Media and UX Lead at Gannett Digital we talk about Web Professional Trends for 2014 including Web and Interactive Design Trends and:

* New and emerging trends for developing and planning for the User Experience
* How the planning process for Web designers has evolved from complex wireframes, paper comps and Visio type resources that developed processes for building monumental structures are moving to more nimble and relevant Agile development processes
* Trends in Interactive prototyping
* How today’s Web designers need to go way beyond the visual and web publishing to more design and develop interactive Web experiences
* Relying more on the code and CSS
* How responsive design and mobile technology has moved the needle and demand for a broader set of skills
* We’ve moved beyond the “fatal five fonts” to more enhanced Typography
* An improved understanding and broader acceptance for a solid understanding of HTML and CSS for Web designers
* How Web and Interactive designers and the technology are pushing the capabilities of today’s browsers
* How the browsers and the technology are catching up

The post Web Professional Trends for 2014 – Web and Interactive Design with Jason Cranford Teague appeared first on Web Professionals.

View full post on Web Professional Minute

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)