keyboard

Better keyboard navigation with progressive enhancement

Keyboard

When building interfaces, it is important to also consider those who can only use a keyboard to use your products. This is a basic accessibility need, and in most cases it isn’t hard to allow for a basic keyboard access. It means first and foremost using keyboard accessible elements for interaction:

  • anchors with a valid href attribute if you want the user to go somewhere
  • buttons when you want to execute your own code and stay in the document

You can make almost everything keyboard accessible using the roving tab index technique, but why bother when there are HTML elements that can do the same?

Making it visual

Using the right elements isn’t quite enough though; you also need to make it obvious where a keyboard user is in a collection of elements. Browsers do this by putting an outline around active elements. Whilst dead useful this has always been a thorn in the side of people who want to control the whole visual display of any interaction. You can remove this visual aid by setting the CSS outline property to none, which is a big accessibility issue unless you also provide an alternative.

By using the most obvious HTML elements for the job and some CSS to ensure that not only hover but also focus states are defined we can make it easy for our users to navigate a list of items by tabbing through them. Shift-Tab allows you to go backwards. You can try it here and the HTML is pretty straight forward.

<ul>
  <li><button>1</button></li>
  <li><button>2</button></li>
  <li><button>3</button></li><li><button>20</button></li>
</ul>

example how to tab through a list of buttons

Using a list gives our elements a hierarchy and a way to navigate with accessible technology that a normal browser doesn’t have. It also gives us a lot of HTML elements to apply styling to. With a few styles, we can turn this into a grid, using less vertical space and allowing for more content in a small space.

ul, li {
  margin: 0;
  padding: 0;
  list-style: none;
}
button {
  border: none;
  display: block;
  background: goldenrod;
  color: white;
  width: 90%;
  height: 30px;  
  margin: 5%;
  transform: scale(0.8);
  transition: 300ms;
}
button:hover, button:focus {
  transform: scale(1);
  outline: none;
  background: powderblue;
  color: #333;
}
 
li {
  float: left;
}
 
/* 
  grid magic by @heydonworks 
  
*/
 
li {
  width: calc(100% / 4);
}
li:nth-child(4n+1):nth-last-child(1) {
  width: 100%;
}
li:nth-child(4n+1):nth-last-child(1) ~ li {
  width: 100%;
}
li:nth-child(4n+1):nth-last-child(2) {
  width: 50%;
}
li:nth-child(4n+1):nth-last-child(2) ~ li {
  width: 50%;
}
li:nth-child(4n+1):nth-last-child(3) {
  width: calc(100% / 4);
}
li:nth-child(4n+1):nth-last-child(3) ~ li {
  width: calc(100% / 4);
}

The result looks pretty fancy and it is very obvious where we are in our journey through the list.

tabbing through a grid item by item

Enhancing the keyboard access – providing shortcuts

However, if I am in a grid, wouldn’t it be better if I could move in two directions with my keyboard?

Using a bit of JavaScript for progressive enhancement, we get this effect and can navigate the grid either with the cursor keys or by using WASD:

navigating inside a grid of elements using the cursor keys going up, down, left and right

It is important to remember here that this is an enhancement. Our list is still fully accessible by tabbing and should JavaScript fail for any of the dozens of reasons it can, we lost a bit of convenience instead of having no interface at all.

I’ve packaged this up in a small open source, vanilla, dependency free JavaScript called gridnav and you can get it on GitHub. All you need to do is to call the script and give it a selector to reach your list of elements.

<ul id="links" data-amount="5" data-element="a">
  <li><a href="#">1</a></li>
  <li><a href="#">2</a></li><li><a href="#">25</a></li>
</ul>
 
<script src="gridnav.js"></script>
<script>
  var linklist = new Gridnav('#links');
</script>

You define the amount of elements in each row and the keyboard accessible element as data attributes on the list element. These are optional, but make the script faster and less error prone. There’s an extensive README explaining how to use the script.

How does it work?

When I started to ponder how to do this, I started like any developer does: trying to tackle the most complex way. I thought I needed to navigate the DOM a lot using parent nodes and siblings with lots of comparing of positioning and using getBoundingClientRect.

Then I took a step back and realised that it doesn’t matter how we display the list. In the end, it is just a list and we need to navigate this one. And we don’t even need to navigate the DOM, as all we do is go from one element in a collection of buttons or anchors to another. All we need to do is to:

  1. Find the element we are on (event.target gives us that).
  2. Get the key that was pressed
  3. Depending on the key move to the next, previous, or skip a few elements to get to the next row

Like this (you can try it out here):

moving in the grid is the same as moving along an axis

The amount of elements we need to skip is defined by the amount of elements in a row. Going up is going n elements backwards and going down is n elements forwards in the collection.

diagram of navigation in the grid

The full code is pretty short if you use some tricks:

(function(){
  var list = document.querySelector('ul');
  var items = list.querySelectorAll('button');
  var amount = Math.floor(
        list.offsetWidth / 
        list.firstElementChild.offsetWidth
      );
  var codes = {
    38: -amount,
    40: amount, 
    39: 1,
    37: -1
  };
  for (var i = 0; i < items.length; i++) {
    items[i].index = i;
  }
  function handlekeys(ev) {
    var keycode = ev.keyCode;
    if (codes[keycode]) {
      var t = ev.target;
      if (t.index !== undefined) {
        if (items[t.index + codes[keycode]]) {
          items[t.index + codes[keycode]].focus();
        }
      }
    }
  }
  list.addEventListener('keyup', handlekeys);
})();

What’s going on here?

We get a handle to the list and cache all the keyboard accessible elements to navigate through

  var list = document.querySelector('ul');
  var items = list.querySelectorAll('button');

We calculate the amount of elements to skip when going up and down by dividing the width of the list element by the width of the first child element that is an HTML element (in this case this will be the LI)

  var amount = Math.floor(
        list.offsetWidth / 
        list.firstElementChild.offsetWidth
      );

Instead of creating a switch statement or lots of if statements for keyboard handling, I prefer to define a lookup table. In this case, it is called codes. They key code for up is 38, 40 is down, 39 is right and 37 is left. If we now get codes[37] for example, we get -1, which is the amount of elements to move in the list

  var codes = {
    38: -amount,
    40: amount, 
    39: 1,
    37: -1
  };

We can use event.target to get which button was pressed in the list, but we don’t know where in the list it is. To avoid having to loop through the list on each keystroke, it makes more sense to loop through all the buttons once and store their index in the list in an index property on the button itself.

  for (var i = 0; i < items.length; i++) {
    items[i].index = i;
  }

The handlekeys() function does the rest. We read the code of the key pressed and compare it with the codes lookup table. This also means we only react to arrow keys in our function. We then get the current element the key was pressed on and check if it has an index property. If it has one, we check if an element exist in the collection that is in the direction we want to move. We do this by adding the index of the current element to the value returned from the lookup table. If the element exists, we focus on it.

  function handlekeys(ev) {
    var keycode = ev.keyCode;
    if (codes[keycode]) {
      var t = ev.target;
      if (t.index !== undefined) {
        if (items[t.index + codes[keycode]]) {
          items[t.index + codes[keycode]].focus();
        }
      }
    }
  }

We apply a keyup event listener to the list and we’re done 🙂

  list.addEventListener('keyup', handlekeys);

If you feel like following this along live, here’s a quick video tutorial of me explaining all the bits and bobs.

The video has a small bug in the final code as I am not comparing the count property to undefined, which means the keyboard functionality doesn’t work on the first item (as 0 is falsy).

View full post on Christian Heilmann

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Keyboard events in Firefox OS TV: Part 2

Implementation details for keyboard events

In our introductory post, Keyboard events in Firefox OS TV, we described four keyboard event scenarios triggered by the Info key on a Smart TV remote: SYSTEM-ONLY, SYSTEM-FIRST, APP-CANCELLED, AND APP-FIRST. We explained how these keyboard events are activated, described the default sequence of events, and explored the iframe structure in Firefox OS and its embedded browser, in order to understand how developers can implement the four keyboard event scenarios.

Now we’ll take a closer look at each of the four scenarios, complete with example code for each event-handling scenario.

SYSTEM-ONLY

If a keyboard event is categorized as SYSTEM-ONLY, then the desired response is defined in mozbrowserbeforekey*’s event handler. Once this key is pressed, the system receives the mozbrowserbeforekey* event before the key* event is dispatched to an app. In addition, the key* events dispatch is cancelled once the system event handler is called. Now, we need to figure out a way to stop the event dispatch. Fig 5 – Keyboard events in Firefox OS TV shows that the keyboard events are dispatched to the system process, then also to the app process. To stop dispatching events to the the embedded page, event.preventDefault() is a straightforward solution. Event.preventDefault() is used to prevent the default action. The defined default action of the mozbrowserbeforekey* event is to dispatch the key* event. For this reason, by calling event.preventDefault() in mozbrowserbeforekey*’s event handler, key* events won’t be dispatched. The final result will be the same as shown in Fig 7.

f7

[Fig 7]

SYSTEM-FIRST

This is very similar to the implementation of SYSTEM-ONLY. The only difference — it’s not necessary to call event.preventDefault() in mozbrowserbeforekey*’s event handler. Apps are able to handle the key* event after the system finishes processing it.

f8

[Fig 8]

APP-CANCELLED
f9

[Fig 9]

If specific keyboard events are designated for use by apps only, such as those assigned to the four colored keys (shown in Fig 9) on smart TV remotes, then event.preventDefault() will be called in the app’s key* event handler. The event.preventDefault() cannot prevent the mozbrowserafterkey* event from being dispatched to the system, but the property embeddedCancelled of mozbrowserafterkey* will be set to true once the embedded app calls event.preventDefault(). The value of embeddedCancelled lets the system detect whether or not this event has been handled already. If this value is true, the system does nothing.

f10

[Fig 10]

APP-FIRST

The difference between APP-FIRST and APP-CANCELLED is that with APP-FIRST event.preventDefault() will not be called in the app’s event handler. Therefore, the value of embeddedCancelled is false and the system can take over the keyboard event.

f11

[Fig 11]

Sample code

Event handlers
function handleEvent(event) {
  dump("Receive event '" + event.type + "'.");
  // Handle event here.....
};

function handleEventAndPreventDefault(event) {
  dump("Receive event '" + event.type + "'.");
  // Handle event here.....

  // Call preventDefault() to stop the default action.
  // It means that the event is already handled.
  event.preventDefault();
};

function checkAttrAndHandleEvent(event) {
  dump("Receive event '" + event.type + 
       "' with embeddedCancelled equals to '" +
       event.embeddedCancelled + "'.");
  if (!event.embeddedCancelled) {
    // Do something if the event wasn't being handled before!
    // The following code should be executed in APP-FIRST scenario only!
  }
};
SYSTEM-ONLY
  • mozbrowser iframe host page
window.addEventListener('mozbrowserbeforekeydown', handleEventAndPreventDefault);
window.addEventListener('mozbrowserbeforekeyup', handleEventAndPreventDefault);
window.addEventListener('mozbrowserafterkeydown', function() { }); // no use
window.addEventListener('mozbrowserafterkeyup', function() { }); // no use
  • the embedded page
// This function will never be triggered because the preventDefault() is called in mozbrowserbeforekeyXXX's handler.
window.addEventListener('keydown', handleEvent);
window.addEventListener('keyup', handleEvent);
  • Results of keydown related events
Order the embedded page mozbrowser iframe host page Output
1 mozbrowserbeforekeydown Receive event ‘mozbrowserbeforekeydown’.
2 mozbrowserafterkeydown
  • Results of keyup related events
Order the embedded page the host page Output
1 mozbrowserbeforekeyup Receive event ‘mozbrowserbeforekeyup’.
2 mozbrowserafterkeyup
SYSTEM-FIRST
  • mozbrowser iframe host page
window.addEventListener('mozbrowserbeforekeydown', handleEvent);
window.addEventListener('mozbrowserbeforekeyup', handleEvent);
window.addEventListener('mozbrowserafterkeydown', function() { }); // no use
window.addEventListener('mozbrowserafterkeyup', function() { }); // no use
  • the embedded page
window.addEventListener('keydown', handleEvent);
window.addEventListener('keyup', handleEvent);
  • Received results of keydown related events
Order mozbrowser-embedded page mozbrowser iframe host page Output
1 mozbrowserbeforekeydown Receive event ‘mozbrowserbeforekeydown’.
2 keydown Receive event ‘keydown’.
3 mozbrowserafterkeydown
  • Received results of keyup related events
Order the embedded page mozbrowser iframe host page Output
1 mozbrowserbeforekeyup Receive event ‘mozbrowserbeforekeyup’.
2 keyup Receive event ‘keyup’.
3 mozbrowserafterkeyup Receive event ‘mozbrowserafterkeyup’ with embeddedCancelled equals to ‘true’.
APP-CANCELLED
  • mozbrowser iframe host page
window.addEventListener('mozbrowserbeforekeydown', function() { }); // no use
window.addEventListener('mozbrowserbeforekeyup', function() { }); // no use
window.addEventListener('mozbrowserafterkeydown', checkAttrAndHandleEvent);
window.addEventListener('mozbrowserafterkeyup', checkAttrAndHandleEvent);
  • mozbrowser iframe embedded page
window.addEventListener('keydown', handleEventAndPreventDefault);
window.addEventListener('keyup', handleEventAndPreventDefault);
  • Received results of keydown related events
Order the embedded page mozbrowser iframe host page Output
1 mozbrowserbeforekeydown
2 keydown Receive event ‘keydown’.
3 mozbrowserafterkeydown Receive event ‘mozbrowserafterkeydown’ with embeddedCancelled equals to ‘true’.
  • Received results of keyup related events
Order mozbrowser-embedded page mozbrowser iframe host page Output
1 mozbrowserbeforekeyup
2 keyup Receive event ‘keyup’.
3 mozbrowserafterkeyup Receive event ‘mozbrowserafterkeyup’ with embeddedCancelled equals to ‘true’.
APP-FIRST
  • mozbrowser iframe host page
window.addEventListener('mozbrowserbeforekeydown', function() { }); // no use
window.addEventListener('mozbrowserbeforekeyup', function() { }); // no use
// This will be trigger after keydown event is
// dispatched to mozbrowser iframe embedded page
window.addEventListener('mozbrowserafterkeydown', checkAttrAndHandleEvent);
window.addEventListener('mozbrowserafterkeyup', checkAttrAndHandleEvent);
  • mozbrowser iframe embedded page
window.addEventListener('keydown', handleEvent);
window.addEventListener('keyup', handleEvent);
  • Received results of keydown related events
Order mozbrowser-embedded page mozbrowser-iframe host page Output
1 mozbrowserbeforekeydown
2 keydown Receive event ‘keydown’.
3 mozbrowserafterkeydown Receive event ‘mozbrowserafterkeydown’ with embeddedCancelled equals to ‘false’.
  • Received results of keyup related events
Order mozbrowser-embedded page mozbrowser iframe host page Output
1 mozbrowserbeforekeyup
2 keyup Receive event ‘keyup’.
3 mozbrowserafterkeyup Receive event ‘mozbrowserafterkeyup’ with embeddedCancelled equals to ‘false’.

 

 

 

View full post on Mozilla Hacks – the Web developer blog

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Keyboard events in Firefox OS TV

Getting started

The behavior of input events via hardware keys in Firefox OS varies widely from app to app. Early smartphones came with a limited number of keys — Power, Home, Volume up, Volume down — so it was easy for the software to determine an appropriate response for each keypress event. However, Smart TV remotes now come with many hardware keys, and defining the appropriate behavior when a key is pressed has become an important issue on the Firefox OS TV platform. If a hardware key on a smart remote can be used both by apps and by the system, it’s important to determine which response is triggered when the key is pressed.

This post will introduce the challenges of programming a TV remote to manage keyboard events on the Firefox OS Smart TV platform. We’ll classify keyboard events into four scenarios and describe these dispatch scenarios and how they interact. This is the first of two posts about keyboard events for Firefox OS Smart TV. Next week, in Part 2, we’ll drill down into the implementation details and and take a look at sample code for event handling on TV remote.

f1

[Fig 1]

We begin with the ‘Info’ key on a TV remote. Often, it’s used by the hardware to display system information, although it’s possible for an application to use the same key to display app information. When a user presses the key, what action will be shown on screen — system info or app info?

Four keyboard event scenarios

To determine the appropriate behavior when hardware keys are pressed, we start by describing four scenarios for keyboard events.

Scenario Description Event order
SYSTEM-ONLY For keys which should be handled by mozbrowser-iframe-host-page only. system
SYSTEM-FIRST For keys which can be handled by mozbrowser-iframe-host-page first and can then also be handled by mozbrowser-iframe-embedded-page. system->app
APP-CANCELLED For keys which should be handled by the mozbrowser-iframe-embedded-page only. app
APP-FIRST For keys which can be handled by mozbrowser-iframe-embedded-page first and can then also be handled by mozbrowser-iframe-host-page. app->system

[Table 1]

[Fig 2]

[Fig 2]

The mozbrowser-iframe-host-page and mozbrowser-iframe-embedded-page mentioned in Table 1 are illustrated in Fig.2 above. If A.html represents a host page whose source is B.html, then A.html is the mozbrowser-iframe-host-page, and B.html is mozbrowser-iframe-embedded page. mozbrowser uses the non-standard Firefox Browser API, built for the implementation of key features and content experiences in Firefox OS apps. Learn more about mozbrowser on MDN.

Suitable responses for any given keyboard event depend on the scenario. In the case illustrated above, let’s suppose that the ‘Info’ keyboard event is categorized as APP-FIRST and the default action set by system is to show system information. Thus, when we press the ‘Info’ key with app Z in the foreground, there are two possible results:

    1. If app Z has an event handler that tells the ‘Info’ key to show app information, then app information will appear on screen when the user presses the ‘Info’ key on the remote.
    2. If app Z doesn’t set an event handler for the ‘Info’ key, the default action is triggered. That is, the screen will show the system information.

How to implement the four scenarios

To implement the four keyboard event scenarios described above, we’ve introduced four new keyboard events:

      • mozbrowserbeforekeydown – fired before the keydown event
      • mozbrowserafterkeydown – fired after the keydown event
      • mozbrowserbeforekeyup – fired before the keyup event
      • mozbrowserafterkeyup – fired after the keyup event

These four keyboard events are only received by the window that embeds a mozbrowser-iframe.

The keyboard events occur in a specific sequence over time: mozbrowserbeforekeydown, mozbrowserafterkeydown, mozbrowserbeforekeyup, keyup, mozbrowserafterkeyup.

This gives developers a way to implement the four scenarios mentioned above. Conceptually, the scenarios SYSTEM-ONLY, SYSTEM-FIRST and APP-CANCELLED, APP-FIRST can be implemented by setting proper handlers for the mozbrowserbeforekey* and mozbrowserafterkey* events. The SYSTEM-ONLY and SYSTEM-FIRST scenarios can be implemented by setting proper handlers for mozbrowserbeforekey* events and the APP-CANCELLED and APP-FIRST scenarios can be implemented via mozbrowserafterkey* events.

iframe structure in Firefox OS

[Fig 3]

[Fig 3]

To understand how to implement the four scenarios, let’s first take a look at iframe structure in Firefox OS. The outermost iframe in Firefox OS is shell.html. It embeds an in-process iframe which is sourced from system/index.html. The system app (system/index.html) contains several web apps (essentially iframes) which can be in-process (remote=”false”) or out-of-process (remote=”true”). As a result, the relationship of these three layers can be displayed in the following table:

mozbrowser iframe host page mozbrowser iframe embedded page
shell.html system/index.html
system/index.html web apps(essentially iframes)

[Table 2]

Dispatch order of keyboard events

[Fig 4]

[Fig 4]

When a keydown event is sent to some element in a mozbrowser-iframe-embedded-page, the owner of the embedded iframe, i.e., the mozbrowser-iframe-host-page, will receive the mozbrowserbeforekeydown event before the keydown event is sent and the mozbrowserafterkeydown event after the event is sent to the mozbrowser-iframe-embedded-page.

In Gecko, once there is one keydown event with the target in an out-of-process iframe embedded in an HTML document, the keydown event is duplicated to the HTML document as well. The target of this duplicated event is set as the embedded iframe element.

This results in the keyboard event sequence shown in Fig 4. It illustrates all related keydown events and their relationship when a keydown event with a target in a mozbrowser-iframe-embedded-page needs to be dispatched. In brief, events follow this sequence:

      1. Before dispatching any keydown event, the mozbrowserbeforekeydown event is first dispatched to the window of mozbrowser-iframe-host-page.
      2. Then, the original keydown event (with a target in a mozbrowser-iframe-embedded-page) will be duplicated to the mozbrowser-iframe-host-page HTML document. Its target will be set to be the iframe that contains the mozbrowser-iframe-embedded-page.
      3. Next, the original keydown event will be dispatched to its target.
      4. After the original keydown event dispatch is complete, the mozbrowserafterkeydown event will be dispatched to the window of mozbrowser-iframe-host-page.

Notice that the event dispatch process described above follows the DOM tree event flow. Event sequence and event targets can be organized into the following table:

Order event target
1 mozbrowserbeforekeydown window in mozbrowser-iframe-host-page
2 keydown iframe that contains the mozbrowser-iframe-embedded-page in mozbrowser-iframe-host-page
3 keydown original one in mozbrowser-iframe-embedded-page
4 mozbrowserafterkeydown window in mozbrowser-iframe-host-page

[Table 3]

f5

[Fig 5]

The keyboard events mozbrowserbeforekeydown, keydown, and mozbrowserafterkeydown, can be extended to nested mozbrowser iframes, like the iframe structure in Firefox OS described in Table 2. In this case, the mozbrowserbeforekeydown and mozbrowserafterkeydown events will be dispatched to the innermost mozbrowser-iframe-host-page as well as the outer one. Thus, in Firefox OS, mozbrowserkeydown and mozbrowserafterkeydown will be dispatched to the window of system/index.html and the window of shell.html. Fig. 5 illustrates the whole dispatch sequence of related events when a keydown event is dispatched to a web app. The sequence of events is demonstrated in Table 4.

Order event target
1 mozbrowserbeforekeydown window in shell.html
2 mozbrowserbeforekeydown window in system/index.html
3 keydown iframe that contains the web app in system/index.html
4 keydown original one in web app
5 mozbrowserafterkeydown window in system/index.html
6 mozbrowserafterkeydown window in shell.html

[Table 4]

f6

[Fig 6]

Although the keyup event must be fired after keydown, the keydown event and the keyup event are independent of each other. Moreover, the path mozbrowserbeforekeyup, keyup, mozbrowserafterkeyup is independent of the path mozbrowserbeforekeydown, keydown, mozbrowserafterkeydown. Therefore, it’s possible for these two paths to cross each other. The mozbrowserbeforekeyup may arrive before the keydown event.

In Firefox OS, most apps run out-of-process. This means that the app runs on its own process, not on the main process. After dispatching a given key* event (the duplicate) to the system app, it takes time to send the original key* event to the process where the mozbrowser-iframe-embedded-page is located. In a similar manner, after a given key* event is dispatched to the mozbrowser-iframe-embedded-page’s process, time is required to send mozbrowserafterkey* back to the process where the mozbrowser-iframe-host-page is located.

Consequently, the mozbrowserbeforekeyup event may arrive in the main Firefox OS process (where the system app lives), before the keydown event is dispatched to the app’s own process. Common results of the order of the key* events are demonstrated above in Fig. 6. The yellow series represents the keydown path, and the blue series show the keyup path. And yes, these two paths may cross each other.

Next steps

If you’re looking for implementation details for the four scenarios we’ve described, as well as event handler sample code to help with your implementation of keyboard events for smart TV remotes, keep an eye for next week’s post, and stay tuned for more complete documentation arriving very soon on MDN.

View full post on Mozilla Hacks – the Web developer blog

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Adding cursor swipe to the Firefox OS keyboard

In this article we will take a look at how to approach adding features to a core component in the system such as the input keyboard. It turns out it is pretty easy!

Before we start, take a look at this concept video from Daniel Hooper to get an idea of what we want to implement:

Cool, huh? Making such a change for other mobile platforms would be pretty hard or just plain impossible, but in Firefox OS it is quite simple and it will take us less than 50 lines of code.

The plan

Conceptually, what we want to achieve is that when the user swipes her finger on the keyboard area, the cursor in the input field moves a distance and direction proportional to the swiping, left or right.

Since a common scenario is that the user might be pressing a wrong key and would like to slide to a close-by key to correct it, we will only start moving the cursor when the swipe distance is longer than the width of a single key.

Preparing your environment

In order to start hacking Firefox OS itself, you will need a copy of Gaia (the collection of webapps that make up the frontend of Firefox OS) and B2G desktop (a build of the B2G app runtime used on devices where all apps should run as they would on a device).

You can take a look at this previous article from Mozilla Hacks in which we guide you through setting up and hacking on Gaia. There is also a complete guide to setting up this environment at https://wiki.mozilla.org/Gaia/Hacking.

Once you get Gaia to run in B2G, you are ready to hack!

Ready to hack!

Firefox OS is all HTML5, and internally it is composed by several ‘apps’. We can find the main system apps in the apps folder in the gaia repository that you cloned before, including the keyboard app that we will be modifying.
In this post we will be editing only apps/keyboard/js/keyboard.js, which is where
a big chunk of the keyboard logic lives.

We start by initializing some extra variables at the top of the file that will help us keep track of the swiping later.

var swipeStartMovePos = null; // Starting point of the swiping
var swipeHappening = false; // Are we in the middle of swiping?
var swipeLastMousex = -1; // Previous mouse position
var swipeMouseTravel = 0; // Amount traveled by the finger so far
var swipeStepWidth = 0; // Width of a single keyboard key

Next we should find where the keyboard processes touch events. At
the top of keyboard.js we see that the event handlers for touch events are
declared:

var eventHandlers = {
  'touchstart': onTouchStart,
  'mousedown': onMouseDown,
  'mouseup': onMouseUp,
  'mousemove': onMouseMove
};

Nice! Now we need to store the coordinates of the initial touch event. Both onTouchStart and onMouseDown end up calling the function startPress after they do their respective post-touch tasks, so we will take care of storing the coordinates there.

startPress does some work for when a key is pressed, like highlighting the key or checking whether the user is pressing backspace. We will write our logic after that. A convenient thing is that one of the arguments in its signature is coords, which refers to the coordinates where the user started touching, in the context of the keyboard element. So storing the coordinates is as easy as that:

function startPress(target, coords, touchId) {
  swipeStartMovePos = { x: coords.pageX, y: coords.pageY };
  ...

In that way we will always have available the coordinates of the last touch even starting point.

The meat of our implementation will happen during the mousemove event, though. We see that the function onMouseMove is just a simple proxy function for the bigger movePress function, where the ‘mouse’ movements are processed. Here is where we will write our cursor-swiping logic.

We will use the width of a keyboard key as our universal measure. Since the width of keyboard keys changes from device to device, we will first have to retrieve it calling a method in IMERender, which is the object that controls how the keyboard is rendered on the screen:

swipeStepWidth = swipeStepWidth || IMERender.getKeyWidth();

Now we can check if swiping is happening, and whether the swiping is longer than swipeStepWidth. Conveniently enough, our movePress function also gets passed the coords object:

if (swipeHappening || (swipeStartMovePos && Math.abs(swipeStartMovePos.x - coords.pageX) > swipeStepWidth)) {

Most of our logic will go inside that ‘if’ block. Now that we know that swiping is happening, we have to determine what direction it is going, assigning 1 for right and -1 for left to our previously initialized variable swipeDirection. After that, we add the amount of distance traveled to the variable swipeMouseTravel, and set swipeLastMousex to the current touch coordinates:

var swipeDirection = coords.pageX > swipeLastMousex ? 1 : -1;
 
if (swipeLastMousex > -1) {
  swipeMouseTravel += Math.abs(coords.pageX - swipeLastMousex);
}
swipeLastMousex = coords.pageX;

Ok, now we have to decide how the pixels travelled by the user’s finger will translate into cursor movement. Let’s make that half the width of a key. That means that for every swipeStepWidth / 2 pixels travelled, the cursor in the input field will move one character.

The way we will move the cursor is a bit hacky. What we do is to simulate the pressing of ‘left arrow’ or ‘right arrow’ by the user, even if these keys don’t even exist in the phone’s virtual keyboard. That allows us to move the cursor in the input field. Not ideal, but Mozilla is about to push a new Keyboard IME API that will give the programmer a proper API to manipulate curor positions and selections. For now, we will just workaround it:

var stepDistance = swipeStepWidth / 2;
if (swipeMouseTravel > stepDistance) {
  var times = Math.floor(swipeMouseTravel / stepDistance);
  swipeMouseTravel = 0;
  for (var i = 0; i < times; i++)
    navigator.mozKeyboard.sendKey(swipeDirection === -1 ? 37 : 39, undefined);
}

After that we just need to confirm that swiping is happening and do some cleanup of timeouts and intervals initialized in other areas of the file, that because of our new swiping functionality ouldn’t get executed otherwise. We also call hideAlternatives to avoid the keyboard to present us with alternative characters while we are swiping.

swipeHappening = true;
 
clearTimeout(deleteTimeout);
clearInterval(deleteInterval);
clearTimeout(menuTimeout);
hideAlternatives();
return;

The only thing left to do is to reset all the values we’ve set when the user lifts her finger off the screen. The event handler for that is onMouseUp, which calls the function endPress, at the beginning of which we will put our logic:

// The user is releasing a key so the key has been pressed. The meat is here.
function endPress(target, coords, touchId) {
    swipeStartMovePos = null;
    ...
    if (swipeHappening === true) {
        swipeHappening = false;
        swipeLastMousex = -1;
        return;
    }

With this last bit, our implementation is complete. Here is a rough video I’ve made with the working implementation:

You can see the complete implementation code changes on GitHub.

Conclusion

Contributing bugfixes or features to Firefox OS is as easy as getting Gaia, B2G and start hacking in HTML5. If you are comfortable programming in JavaScript and familiar with making web pages, you can already contribute to the mobile operating system from Mozilla.

Appendix: Finding an area to work on

If you already know what bug you want to solve or what feature you want to implement in Firefox OS, first check if it has already been filed in Bugzilla, which is the issue repository that Mozilla uses to keep track of bugs. If it hasn’t, feel free to add it. Otherwise, if you are looking for new bugs to fix, a quick search will reveal many new ones that are sill unassigned. Feel free to pick them up!

View full post on Mozilla Hacks – the Web developer blog

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)