DATA

Backing Up User Data on Firefox OS

The FFOS Backup/Restore Team

Portland State University’s Computer Science degree culminates in a capstone program that matches teams of students with industry sponsors for a six-month software engineering project. We had the privilege of working with Mozilla on an application to back up and restore personal data on Firefox OS. We are:

Ryan Bernstein: Team Lead/History/Demo application.
David Cobbley: Contacts/SMS Messages.
Thomas Guerena: Architecture/Build environment/Media.
Wu Hao: SMS Messages/Space checking.
Kai Jiang: SMS Messages. Space checking.
Nathan Larson: History/System settings.
Ruben Niculcea: Build environment/Media/SMS Messages.
Dean Nida: Media/Contacts/Testing.

From left: Nathan Larson, Jiang Kai, Wu Hao, David Cobbley, Dean Nida, Ruben Niculcea, Thomas Guerena, Ryan Bernstein

From left: Nathan Larson, Jiang Kai, Wu Hao, David Cobbley, Dean Nida, Ruben Niculcea, Thomas Guerena, Ryan Bernstein

The Firefox OS Backup/Restore Library

The purpose of this capstone project was to allow users of Firefox OS to back up and restore their personal data. The OS does not provide any native support for such backups, despite the fact that it is sorely needed; Firefox OS was initially targeted at developing nations, where it runs on inexpensive hardware and may be the user’s primary computing device.

Making a backup application is a difficult task. As developers, we’re unable to anticipate future changes in both the operating system itself and the types of data that users create. Ultimately, we decided that the ephemeral nature of a senior capstone team meant that we would be unable to provide the long-term support necessary to deploy and maintain an application on the app store.

We therefore decided to create a library, rather than a standalone application. Our reasoning was that we would be able to make a stronger impact by providing a foundation for other developers to build their own backup applications on. After six months of development, we had created the Firefox OS Backup/Restore (FFOSBR) library, as well as a small test application to demonstrate its basic capabilities.

Capabilities of FFOSBR

The Firefox OS Backup/Restore (FFOSBR) library is designed to back up media to an SD card. While we had envisioned being able to back up data via USB as well, this would have required the development of a companion application to interpret the data on the receiving PC, which was outside the scope of our project.

The FFOSBR library is capable of backing up and restoring:

  • Photos
  • Music
  • Videos
  • Contacts
  • System Settings

The library can also back up the user’s SMS and MMS messages. However, Firefox OS currently provides no API for adding messages to the phone without actually sending them via SMS or MMS. As such, messages are backed up as JSON objects to a text file on the SD card, but cannot be restored back to the phone.

Implementation

FFOSBR is implemented as a collection of modules. Each type of personal data has a module, but there are additional “helper” modules that are used to hold the library together. Notably, these include a ffosbr.settings module to track application settings (such as which data types have backups enabled) and a ffosbr.history module that tracks backup dates and sizes for each data type.

Data type-specific modules each implement, at minimum, three public functions: backup(), restore(), and clean(). These functions all take a single callback parameter, oncomplete. The callback should be a function that takes two arguments. The first argument will always be the name of the type backed up; any errors that occur will be passed to the oncomplete function in the second argument. Passing the type as the first argument allows us to use a technique that we refer to as patient oncompletes, which will be described shortly.

In addition, there are top-level helper modules — ffosbr.backup(), ffosbr.restore(), and ffosbr.clean() — which iterate over all of the enabled data type-specific modules and invoke their backup(), restore(), or clean() methods, respectively. This is where patient oncompletes come into play. Let’s look at ffosbr.backup() as an example.

ffosbr.backup()

ffosbr.backup() takes three arguments, all of which are callback methods. These are onsuccess, onerror, and oncomplete.

ffosbr.backup() runs each data type’s backup() method asynchronously with a function called callbackManager() as an oncomplete. As mentioned above, each data type’s backup() method calls its oncomplete method with the name of the data type as the first argument so that the callbackManager() can identify it.  As each data type completes, the callbackManager() examines its second argument to determine whether or not an error occurred. If so, it calls onerror(); otherwise, it calls onsuccess(). Only after every data type’s backup() method has completed does the ffosbr.backup() method finish and invoke its own oncomplete callback.

Usage

A function that backs up all types of user data might therefore look something like this:


var successes = [];
var failures = [];

var reportSuccess = function(type) {
  if (type && !successes.includes(type)) {
    alertUser(type + ' saved successfully');
    successes.push(type);
  }
};

var reportError = function(type, error) {
  if (type && !failures.includes(type)) {
    alertUser(type + ' failed');
    failures.push(type);
  }
};

var finished = function() {
  var sitrep = 'SUCCESSES:\n';
  for (var i = 0; i < successes.length; ++i) {
    sitrep += '\t' + successes[i] + '\n';
  }
  sitrep += '\nFAILURES:\n';

  for (i = 0; i < failures.length; ++i) {
    sitrep += '\t' + failures[i] + '\n';
  }

  alert(sitrep);
};

ffosbr.backup(reportSuccess, reportError, finished);

Cleaning and restoring look similar, but with ffosbr.clean() or ffosbr.restore() used in place of ffosbr.backup() on the last line.

Demo Application

In addition to the FFOSBR library itself, we’ve also created a simple demo application to show off some of its capabilities. The home screen displays information about current backup status, and has buttons to back up and restore all enabled types:


2015-08-29-03-38-19
 

Clicking on any of the listed types opens a submenu:


2015-08-30-23-47-11
From here, the user can enable or disable backups for that data type. They can also restore data of that type individually, or clear backups of that type from the SD card’s backup directory.

Do keep in mind that our primary product was the library itself; as a demo, the application is relatively bare-bones. However, it does provide the capability for users to back up and restore their media, contacts, and system settings to an SD card.

Future Improvements

Unfortunately, not all backups are entirely nondestructive. Music, pictures, and videos work on a per-file basis. Thus, backing up or restoring data doesn’t remove the data already on the phone or SD card. However, contacts, messages, and system settings are backed up to specific files in the backup directory on the SD card. These files are overwritten with each backup. In the future, we can address this by appending a timestamp to the filenames so that each backup creates a new file.

The contacts module also currently runs synchronously. ffosbr.contacts.backup() calls a function that gets contacts from the SIM card, which in turn calls a function that gets contacts from the device. Ideally, these should be two entirely separate operations which back up to entirely separate files, since it will also allow us to restore SIM contacts back to the SIM card and device contacts back to the device. We didn’t devise our patient oncomplete strategy until after the contacts module had been written, but employing this would allow us to back up SIM and device contacts in parallel to separate files.

Installing Firefox OS Backup/Restore

The FFOSBR project can be found on GitHub at github.com/autonome/FFOSBR. If you are a Firefox OS developer or technical user, take a look and let us know what you think.

View full post on Mozilla Hacks – the Web developer blog

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

WebRTC Data Channels for Great Multiplayer

WebRTC is getting great press lately for it’s amazing applications in voice and video communication. But did you know that WebRTC also has support for peer-to-peer data? Below I’ll talk about the ‘what’ and ‘how’ of data channels, and then I’ll show you how we’re using them in BananaBread to support peer-to-peer multiplayer.

5-player BananaBread with WebRTC, filmed at Mozilla Toronto on a Friday afternoon.

Browser caveats

If you want to use WebRTC data channels, you’ll need the latest Firefox Nightly or Chrome Canary. BananaBread multiplayer requires binary message support for WebRTC data channels, which hasn’t landed just yet in Chrome.

What is a data channel?

A WebRTC data channel lets you send text or binary data over an active peer connection. Data channels come in two flavors. Reliable channels will guarantee that messages you send have arrived at the other peer. If you send multiple reliable messages, they will be received in the same order. You can also send unreliable message that are not guaranteed to arrive in the order they were sent, or they may not arrive at all. These are analogous to TCP and UDP.

Creating and using data channels

Unfortunately, setting up peer connections and creating data channels requires a non-trivial amount of work. It’s very reasonable to rely on a library to handle these things, as well as abstract away some of the implementation details between Firefox and Chrome.

The library used in the examples here is p2p. It provides a simple API for establishing peer connections and setting up streams and data channels. There’s also a broker server component and a hosted broker you can use instead of setting one up for yourself.

Using p2p

You can start with a simple web page. Something like this will do:

<!DOCTYPE HTML>
<html>
<body>
</body>
<script src="http://raw.github.com/js-platform/p2p/develop/client/wrtcp.js"></script>
</html>

In a simple configuration, one peer acts as host and listens for other peers that want to connect.

/* This is the address of a p2p broker server, possibly the example one at http://mdsw.ch:8080 */
var broker = "http://";
 
/* We'll use this to store any active connections so we can get to them later. */
var connections = {};
 
/* This creates an object that will handle let us listen for incoming connections. */
var peer = new Peer(broker, {video: false, audio: false});
 
/* This is invoked whenever a new connection is established. */
peer.onconnection = function(connection) {
  connections[connection.id] = connection;
 
  connection.ondisconnect = function() {
    delete connections[connection.id];
  };
 
  connection.onmessage = function(label, msg) {
    console.log(msg.data);
  };
};
 
/* This is called when your peer has received a routing address from the broker.
   The route is what lets other peers send messages through the broker that are used to
   establish the peer-to-peer connection. */
peer.onroute = function(route) {
  console.log(route);
}
 
/* This tells the broker that this peer is interested in hosting. */
peer.listen();

The connection object pass into onconnection comes with two data channels, helpfully labelled reliable and unreliable. The label, along with the data, is passed to onmessage whenever that connection receives a message.

If your peer is hosting, it’s handy to capture the routing address assigned by the broker. Another peer needs both the broker URL and the route to connect to your peer.

Finally, the connection object also exposes the local and remote streams, in case you want to send video or audio as well:

  connection.streams['local'];
  connection.streams['remote'];

If your peer is connecting to another peer, the code is the same as above except that instead of calling listen you should:

  /* Call this with the routing address that the host received from the broker. */
  peer.connect();

Sockets for Emscripten

In case you’re unfamiliar with Emscripten, the important thing to know is that it compiles C++ libraries and programs to JavaScript, allowing them to run in your browser. This is exactly what we used to turn Sauerbraten into BananaBread.

Sauerbraten has built-in multiplayer support that relies on the POSIX sockets that work very differently from WebRTC peer connections. C++ programs that use sockets expect to communicate with remote hosts by giving an IP address and a port number to the sockets API, along with a buffer containing some arbitrary data. BananaBread in particular only makes four kinds of API calls: socket, bind, recvmsg, and sendmsg.

Each time socket is called, we create a new JS object to hold an address, a port, and a message queue. The queue is important because we will need to hold onto messages that arrive from a data channel so they can be handled later by the application, which will call recvmsg. There’s also some space to build a small header that we will use for sending messages.

Since we’re using the same p2p library from above, the code to create a new Peer is identical except for the event handlers. Here’s the code for onconnection:

peer.onconnection = function(connection) {
  var addr;
  // Assign 10.0.0.1 to the host
  if(route &amp;&amp; route === connection['route']) {
    addr = 0x0100000a; // 10.0.0.1
  } else {
    addr = Sockets.addrPool.shift();
  }
  connection['addr'] = addr;
  Sockets.connections[addr] = connection;
 
  connection.ondisconnect = function() {
    // Don't return the host address (10.0.0.1) to the pool
    if(!(route &amp;&amp; route === Sockets.connections[addr]['route']))
      Sockets.addrPool.push(addr);
    delete Sockets.connections[addr];
  };
 
  connection.onmessage = function(label, message) {
    var header = new Uint16Array(message, 0, 2);
    if(Sockets.portmap[header[1]]) {
      /* The queued message is retrived later when the program calls recvmsg. */
      Sockets.portmap[header[1]].inQueue.push([addr, message]);
    }
  }

Sockets.addrPool is a list of available IP addresses that we can assign to new connections. The address is used to find the right active connection when the C++ program wants to send or receive data.

socket: function(family, type, protocol) {
  var fd = Sockets.nextFd ++;
  Sockets.fds[fd] = {
    addr: undefined,
    port: undefined,
    inQueue: [],
    header: new Uint16Array(2),
  };
  return fd;
};

Bind is invoked directly when a program wants to listen on a given port, and indirectly when sendmsg is used with an unbound socket (so that recvmsg can be called on the socket and the remote host can send a reply). In the latter case we can give the socket any unused port. We don’t need to worry about the IP address here.

bind: function(fd, addr, addrlen) {
  var info = Sockets.fds[fd];
  if (!info) return -1;
  if(addr) {
    /* The addr argument is actually a C++ pointer, so we need to read the value from the Emscripten heap. */
    info.port = _ntohs(getValue(addr + Sockets.sockaddr_in_layout.sin_port, 'i16'));
  } else {
    /* Finds and returns an unused port. */
    info.port = _mkport();
  }
 
  /* We might need to pass the local address to C++ code so we should give it a meaningful value. */
  info.addr = 0xfe00000a; // 10.0.0.254
 
  /* This is used to find the socket associated with a port so we can deliver incoming messages. */
  Sockets.portmap[info.port] = info;
};

For sendmsg, we need find the socket associated with the given IP address. We also need to prepend a small header onto the message buffer that contains the destination port (so the remote host can deliver the message) and the source port (so the remote host can send a reply to the message). Recvmsg is very similar to sendmsg.

(Note that the code for reading and writing data to the msg argument is omitted because it’s quite dense and
doesn’t add very much.)

sendmsg: function(fd, msg, flags) {
  var info = Sockets.fds[fd];
  if (!info) return -1;
 
  /* Here's where we bind to an unused port if necessary. */
  if(!info.port) {
    bind(fd);
  }
 
  /* The next three lines retrieve the destination address and port from the msg argument. */
  var name = {{{ makeGetValue('msg', 'Sockets.msghdr_layout.msg_name', '*') }}};
  var port = _ntohs(getValue(name + Sockets.sockaddr_in_layout.sin_port, 'i16'));
  var addr = getValue(name + Sockets.sockaddr_in_layout.sin_addr, 'i32');
 
  var connection = Sockets.connections[addr];
  if (!(connection &amp;&amp; connection.connected)) {
    /* Emscripten requires that all socket operations are non-blocking. */
    ___setErrNo(ERRNO_CODES.EWOULDBLOCK);
    return -1;
  }
 
  /* Copy the message data into a buffer so we can send it over the data channel. */
  var bytes = new Uint8Array();
 
  info.header[0] = info.port; // Source port
  info.header[1] = port; // Destination port
 
  /* Create a new array buffer that's big enough to hold the message bytes and the header. */
  var data = new Uint8Array(info.header.byteLength + bytes.byteLength);
 
  /* Copy the header and the bytes into the new buffer. */
  buffer.set(new Uint8Array(info.header.buffer));
  buffer.set(bytes, info.header.byteLength);
 
  connection.send('unreliable', buffer.buffer);
};
recvmsg: function(fd, msg, flags) {
  var info = Sockets.fds[fd];
  if (!info) return -1;
 
  /* There's no way to deliver messages to this socket if it doesn't have a port. */
  if (!info.port) {
    assert(false, 'cannot receive on unbound socket');
  }
 
  /* Similar to sendmsg, if there are no messages waiting we return instead of blocking. */
  if (info.inQueue.length() == 0) {
    ___setErrNo(ERRNO_CODES.EWOULDBLOCK);
    return -1;
  }
 
  var entry = info.inQueue.shift();
  var addr = entry[0];
  var message = entry[1];
  var header = new Uint16Array(message, 0, info.header.length);
  var bytes = new Uint8Array(message, info.header.byteLength);
 
  /* Copy the address, port and bytes into the msg argument. */
 
  return bytes.byteLength;
};

What’s next

Both the p2p library and sockets for Emscripten were made to support multiplayer BananaBread, so they’re both missing features that would be useful for building other applications. Specifically,

  • Add support for peer-based brokering in the p2p library (so connected peers can broker new connections for each other)
  • Add support for connection-oriented and reliable webrtc-based sockets in Emscripten

If you’re building something cool with this, or you’d like to and you have questions, feel free to ask me on Twitter or in #games on irc.mozilla.org.

View full post on Mozilla Hacks – the Web developer blog

VN:F [1.9.22_1171]
Rating: 10.0/10 (2 votes cast)
VN:F [1.9.22_1171]
Rating: +2 (from 2 votes)