Researchers at Facebook developed a predictive motion model for marker-based hand tracking that could enable some typists to match their speed and accuracy with a physical keyboard while only tapping their fingers on a flat surface.
I’ve emphasized some of the most notable bits announced by Facebook in a blog post today:
To support touch typing without a physical keyboard — and without the benefit of haptic feedback from individual keys — the team had to make sense of erratic typing patterns. They adopted statistical decoding techniques from automatic speech recognition, and where speech recognition uses an acoustic model to predict phonemes from audio frames they instead use a motion model to predict keystrokes from hand motion. This, along with a language model, predicts what people intended to type despite ambiguous hand motion. Using this new method, typists averaged 73 words per minute with a 2.4% uncorrected error rate using their hands, a flat surface, and nothing else, achieving similar speed and accuracy to the same typist on a physical keyboard.
This surprising result led researchers to investigate why hand tracking was more effective than other physical methods, like tapping on a tablet. The team discovered hand tracking is uniquely good at isolating individual fingers and their trajectories as they reach for keys — information that is missing from capacitive sensing on tablets and smartphones today.
You can see in the video below a person with markers on their hands with external tracking cameras:
I asked a Facebook representative if Quest 2’s XR2 processor might enable any specific features for hand tracking, and if the marker-based research here might transfer to a markerless system as seen with Quest’s hand tracking.
“This marker-based hand tracking system is still purely in the research phase and not currently on our product roadmap — it’s one of several text input research methods FRL Research is exploring, including the EMG-based approach Michael Abrash showed last month at Facebook Connect,” a spokesperson wrote in an email. “For Quest 2, our current focus for text input is tracked keyboard support we introduced at Connect as part of Infinite Office.”
We found multiple text input methods on Oculus Quest 2 in the Oculus Browser prior to a launch software update. There was a mode that would let you tap out letters on your phone and send them to your Quest, as well as voice dictation. Facebook frequently rolls out experimental features for its standalone VR headsets as opt-in testing betas for a period before improving performance and finalizing the feature more broadly, with Oculus Link and Hand Tracking itself being the most significant examples. Right now, if you connect a Bluetooth keyboard and don’t use the tracked controllers, Quest gets confused about whether you are typing input on a physical keyboard or pinching to select a letter on a floating virtual keyboard with hand tracking. That’s likely to change with the forthcoming Infinite Office update and a tracked keyboard.
So while typing on any flat surface is just one of several methods being explored for text entry in AR and VR headsets by Facebook, the company did note the research was “a major step towards FRL Research’s goal of using hand-tracking to let you type 70 WPM on any surface.”
