Starting this week with the launch of hand tracking, Oculus Quest owners no longer need controllers to operate the standalone VR headset.
UPDATE Dec 10: v12 is now rolling out with this feature available!
Facebook calls the feature “Hand Tracking” and it uses the on-board cameras to identify the movements of your fingers and hand without held controllers. It will be available in the ‘Experimental’ section of the Settings in update v12, which is rolling out this week.
This is a firmware update, no new hardware is required. In fact, less hardware is required. For the first time, Quest owners will be able to demonstrate some basic functionality of the standalone headset to friends and family for the holidays in 2019 without Oculus Touch controllers.
The company first announced this feature at Oculus Connect 6 in September, stating that it would arrive in ‘2020’. But in a holiday season surprise they’ve managed to ship it early — for system apps at least.
This isn’t the first headset to support this feature. HTC’s PC-based Vive Cosmos VR headset also supports optical finger tracking. Doing this on a standalone headset, however, is significantly more difficult, and reports of the Cosmos finger tracking quality have not been great.
Quest System Apps At First
The only apps which will support Hand Tracking this week are a handful of system apps. So you’ll have to wait a bit longer for some of your favorite games and experiences to support the feature.
- The main Oculus menu, including the Store and Library
- Oculus Browser (the built in web browser)
- Oculus TV (for both 2D content and 360 videos)
This means you can use Quest as an immersive media viewer or portable cinema without picking up or bringing along the Touch controllers.
We don’t know when Facebook will launch Elixir, an introductory experience used to show off the feature to media at Oculus Connect 6.
SDK Available Next Week
Next week, Facebook plans to release the software development kit (SDK) for Hand Tracking, so that any Quest developer can support the feature in their app or game.
Oculus Avatars is targeted to support finger tracking in 2020, which will allow developers to add hands matching the user’s actual skin tone by default.
We don’t yet know which developers will support Hand Tracking, but we’d expect most social and media apps to do so. We’ll start a list of Quest apps with plans for hand tracking support, so please email tips@uploadvr.com if you’re a VR developer embracing the feature. Bigscreen’s CEO already hinted at support and we confirmed that Virtual Desktop from Guy Godin will be working on the feature when he gets access to the SDK.
Facebook expects the first third party apps to start supporting the feature in 2020.
Keep in mind that it is likely only some games will be adding Hand Tracking support. Many games require buttons, analog sticks, triggers, or haptic feedback. Additionally, based on our experience at Oculus Connect 6, the tracking quality likely wouldn’t be suitable for advanced two hand interactions.
A Long Researched, Challenging Feature
The first indication of Facebook’s interest in finger tracking for VR was revealed in late 2014. The company acquired startup Nimble VR, composed of four veterans of hand tracking technology. But upon this acquisition, Oculus was clear that this technology “may not even be used in the CV2 or CV3”.
That’s likely because Nimble’s tech wasn’t just software, but a dedicated depth sensor. Depth sensors deliver excellent finger tracking such as that found in the new HoloLens 2 AR headset. But these dedicated sensors can add cost and weight to headsets when reducing both is key to broader adoption of the technology. Delivering high quality finger tracking on regular cameras, as is the case on Quest, is a different level of challenge. Its implementation on Quest using the same hardware employed for other kinds of tracking is likely a competitive advantage for Facebook and something likely to be more broadly adopted in future products from the company.
In 2016 at Oculus Connect 3, Chief Scientist Michael Abrash made a range of predictions about VR in the year 2021. He noted that finger tracking could be done perfectly with markers on gloves (and showed this), but not yet directly. However, he predicted that by 2021, it would be possible:
With the roll-out of this feature, then, Facebook will have essentially beaten Abash’s prediction by a year. In 2018 at Facebook’s annual F8 conference the company showed high quality gloveless, markerless finger tracking:
An Oculus representative explained on stage that this “breakthrough” was made possible by Facebook’s heavy investment in machine learning (“AI”) research. They use the kind of glove system described by Abrash in 2016 to train a convolutional neural network to recognize hand poses. This is likely how the feature was made possible.
Far From Perfect
As we noted in our hands-on at Oculus Connect 6, Quest’s finger tracking is far from perfect:
“If I put my fists together my hands could disappear. If I turned my head to the side and then looked back at my hands it would take a second for them to pop back into existence at a spot where I expected them to be. If I moved my hands too fast they could disappear. If I touched my pointer fingers tip to tip their virtual representations still showed a slight gap between them before quickly correcting themselves to a more believable position. All of these I would characterize as momentary lapses during deliberate attempts to break the tracking rather than just enjoying the things for me to do in the fantasy demo world.“
Unless Facebook made dramatic improvements in the past three months, you won’t have the same tracking robustness you get with the Touch controllers. The controllers contain an accelerometer and a constellation of infrared LEDs designed specifically for tracking in a range of conditions.
But finger and hand tracking should still be good enough for social VR, media consumption, and some very casual games designed specifically with the feature’s limitations in mind. Facebook plans to “add new features and functionality to improve the experience of hand tracking on Quest in 2020.”
Like the company did with Touch controller tracking, Facebook will likely improve the finger tracking quality and robustness over time.
At Oculus Connect 6, Engineering Manager Robert Wang explained this current iteration “only scratched the surface of what we can do with hand tracking”. While he was careful not to give a roadmap, he did state that Facebook would “love to be able to” reliably track two handed interactions with regular cameras.
It will certainly be interesting to see what developers do with this feature, and where Facebook will take it in future.
