Researchers at Facebook Reality Labs came up with a virtual keyboard for controller-free hand tracking in VR or AR called PinchType.
Text input in virtual reality is currently considered an unsolved problem. When seated, PC VR users are unable to see their PC’s keyboard, and in room scale experiences it’s inconvenient to access. Standalone VR users may not have a physical keyboard at all.
Facebook’s Oculus Quest standalone headset got ‘experimental’ support for controller-free hand tracking back in December. System updates over past months have improved the quality of this tracking somewhat.
However, given that the technology relies on cameras on the headset without depth sensors, it isn’t currently robust enough for high speed typing on a virtual keyboard.
Three researchers at Facebook Reality Labs are presenting PinchType- progress towards a solution to this text entry problem. It works by assigning each finger a group of letters from the standard QWERTY keyboard. When you pinch that finger against your thumb, that letter group is entered.
As you type the word, a language algorithm determines what word you were intending to type, based on each letter group and previous context. The researchers didn’t mention whether this is a traditional algorithm or using machine learning. Facebook is considered a world leader in Natural Language Processing.
The researchers say PinchType enables an average typing speed of just over 12 words per minute (WPM). For comparison, the average for a QWERTY keyboard is said to be around 40 WPM, or 35 WPM for a modern smartphone.
Currently, text entry on Quest works by selecting individual letters from a virtual floating QWERTY keyboard, one at a time. In hand tracking mode, this makes typing extremely slow.
There are no indications that PinchType is planned to be used in current Oculus software. Facebook Reality Labs often presents research that isn’t intended for consumer use. That said, this does seem like it would be an improvement over the current software.
One day a text input method as fast as a keyboard may require higher fidelity hand tracking and realtime 3D room sensing to project virtual keyboards onto real surfaces. But for now, and for room scale, ideas like PinchType might be able to make hand tracking text entry somewhat usable.
