The Oculus Quest is getting controller-free finger tracking in “early 2020”.
This will allow developers to create apps where you can interact directly with your hands instead of the Touch controllers. Existing apps could also be updated with this functionality, though this will only be suitable for some, such as social VR or passive entertainment, due to the lack of input or haptic feedback.
This confirms our speculative report from February of this year, when we reached out to John Carmack to find out the prospects of this feature arriving.
HTC’s new Vive Cosmos PC VR headset also supports optical finger tracking, although reports of their algorithm’s tracking quality on the Vive Pro have been mixed at best.
A Long Researched, Challenging Feature
The first indication of Facebook’s interest in finger tracking for VR was revealed in late 2014. The company acquired startup Nimble VR, composed of four veterans of hand tracking technology. But upon this acquisition, Oculus was clear that this technology “may not even be used in the CV2 or CV3”.
That’s likely because Nimble’s tech wasn’t just software, but a dedicated depth sensor. Depth sensors deliver excellent finger tracking such as that found in the new HoloLens 2 AR headset. But the high end offerings are costly and the low end ones are bulky and relatively heavy, adding weight at the worst possible area of a headset (directly in front).
Delivering high quality finger tracking on regular cameras is an entirely different level of challenge. However, if it can be done it allows finger tracking to be added at no extra hardware cost to headsets which already use cameras for other tracking.
In 2016 at Oculus Connect 3, Chief Scientist Michael Abrash made a range of predictions about VR in the year 2021. He noted that finger tracking could be done perfectly with markers on gloves (and showed this off), but not yet directly. However, he predicted that by 2021, it would be possible:
If this feature arrives this year, Facebook will have beaten Abash’s prediction by a year.
In 2018 at Facebook’s annual F8 conference the company finally showed off high quality gloveless, markerless finger tracking:
An Oculus representative explained on stage that this “breakthrough” was made possible by Facebook’s heavy investment in machine learning (“AI”) research. They use the kind of glove system described by Abrash in 2016 to train a convolutional neural network to recognize hand poses. This is likely how the feature was made possible.
The Potential
When people put on a VR headset, one of the first things they do is reach out with their hands. Tracked controllers offer a basic representation of our hands. They’re also very suited for gaming. But they don’t track the vast majority of finger motion and the very act of holding them restricts those motions too.
In non-interactive VR experiences and social VR, controllers often feel more like a chore than a help. If we could enter these experiences by just putting on a headset and seeing our real hands, this reduction of friction would be a welcome improvement.
Enterprise customers using VR for tasks like architecture visualization often forgo controllers entirely due to this friction. Gaming-style controllers are simply much less appealing to non-gamers than directly seeing your hands.
In social VR the ability to gesticulate freely with your real hands adds to immersion and increases social presence. I’ve spent a lot of time in platforms like AltSpaceVR- Leap Motion is by far my preferred method of input.
We’ll be closely watching what VR developers do with this feature once it releases.
