Unexplored avenues of optimizing layouts for gesture typing

[devycarol]

This post is a discussion of this research presentation vid on optimizing keyboard layouts for English gesture typing.

This is of particular interest to me as someone who's been working on a gesture typing layout for a few years now. I'm making this post as a knowledge well for potential future work, perhaps even a research project down the line if there ends up being enough depth to explore here.

As discussed in the vid, QWERTY performs poorly as a gesture typing keyboard. There are two big metrics we care about: speed and accuracy. Speed is how long the gesture paths are on average, we want to shorten these. Accuracy is how often the gesture engine correctly guesses the word you're trying to gesture. As with PC alt layouts the learning curve for changing your layout is steep, although not as bad when you don't need to rearrange your keycaps or master blind touch typing. As such, alt layout design should care about fostering learnability so users can catch up to their QWERTY speed and accuracy sooner than later.

Quite interestingly, their research discovered layouts which score highly in "QWERTY similarity" that have negligible difference in speed or clarity to layouts optimized purely to those metrics; they still improve significantly over QWERTY. That's an important point, though I take some issue with how they measure QWERTY similarity.

They use Manhattan distance of keys to their original QWERTY position, with the top row arbitrarily right-shifted to fit the standard keyboard layout shape into a square grid. But anecdotally, I find that vertical space is far more significant than horizontal. I believe this to be the case even if we ignore how keys are taller than they are wide (in phone portrait mode). Rows have 7–10 keys while "columns" only have 3, so swapping column neighbors will change gesture shapes far more drastically than swapping row neighbors. I'd even say a letter moving two or three spaces within a row is a less significant change than moving up or down one. That said, I'm not sure what an alternative learnability metric should look like.

I think they also failed to consider a really important metric: gesture comfort. We typically measure speed as having short path lengths, but comfort affects how quickly you can actually actually complete that movement with your thumb. It also matters in its own right, you'll get tired of typing faster if the layout is less comfortable. This is actually my main reason for using Dvorak on the PC, the comfort benefits are more important than any speed increase.

With phones, your thumb has a particular resting place where it likes to hover. This will vary depending on the shape of the user's hand, the shape of the device, how they hold it, which hand is their dominant one, etc. (BTW, does anyone do any gestures with their non-dominant hand? I've never thought of doing this.) For me, this is where my thumb likes to hover:

Movement is also a comfort concern. This basically doesn't matter for tablets where you can freely glide your middle finger or stylus pen, but when using a phone I find motions involving the base thumb joint much more comfortable than ones where you mainly use the tip joint. So for me (right-handed), it's ideal for gesture patterns to resemble the above picture, whereas patterns like this are less comfortable:

I wonder if others feel the same?

So if we also want to optimize for gesture comfort, here are some pointers.

• More common letters should be around the comfortable thumb zone, less common letters can be in the corners.
  • If my assumptions are correct, this means slightly toward the dominant-hand side of the screen?
• An "equivalent" left handed layout can be found by just mirroring a right-handed one, though it will have different QWERTY similarity properties
• Common digraphs should be along the more comfortable paths, gestures with uncomfortable paths should be avoided

As an aside, I'm actually quite interested in looking at the background data from the HeliBoard gesture dataset as a frequency analysis of words typed on phones, though it's a flawed dataset for that specific purpose. It's sampled from the community of this particular keyboard program, we can assume this sample is disproportionately "techy", male, online. And custom dictionary words are excluded IIRC. Are there any other datasets that analyze word frequency on phones specifically? Hard to imagine an ethical collection method for that lol.