Comments made by Oculus CTO John Carmack at Oculus Connect 5 and on Twitter suggest that the upcoming Oculus Rift S cameras could support finger tracking.
To be clear, this doesn’t mean the software to do so would be ready by the time Rift S launches. But it does mean the headset could one day recieve the feature as a software update in future- the cameras are seemingly suitable.
Rift S Onboard Cameras
Rift S was first revealed in a TechCrunch report in October. The report revealed the headset would be an iterative update, increasing resolution and changing to the same inside-out tracking system as the upcoming Oculus Quest VR console.
Earlier this month we confirmed the TechCrunch report by discovering references in the Oculus PC software code to a ‘Rift S’ with onboard cameras.
Carmack’s Comments
During 2018’s Oculus Connect 5 conference, Oculus CTO John Carmack briefly spoke about a project to bring finger tracking to Oculus Quest:
This seemed to suggest that the main limitation for finger tracking on the headset was the power & compute limitations of the mobile platform. He stated that Quest can be used as a platform to research finger tracking for future headsets with “the power and ability to do real time”.
The TechCrunch report and our findings suggest that Rift S will feature the same Insight cameras as Quest. So given the enormous relative power of a PC, shouldn’t Rift S be capable of finger tracking?
To verify we weren’t misunderstanding his comments, we reached out to Carmack on Twitter. Here’s how he responded:
Carmack doubled down on stressing the power limitations of mobile as the main barrier. When asked about the featue in regards to PC, he confirmed it was possible. Requiring a reserved CPU core or two rules out this tech being usable in CPU-intensive apps. But it would be entirely possible for developers of apps suiting finger tracking to optimize for this limitation.
These comments aren’t indicative of the feature actually being available at launch- or ever. But what they do suggest is that the camera hardware is suitable and PCs are capable.
A Long Researched, Challenging Feature
The first indication of Facebook’s interest in finger tracking for VR was revealed in late 2014. The company acquired startup Nimble VR, composed of four veterans of hand tracking technology. But upon this acquisition, Oculus was clear that this tecnology “may not even be used in the CV2 or CV3”.
That’s likely because Nimble’s tech wasn’t just software, but a dedicated depth sensor. Depth sensors deliver excellent finger tracking such as that found in the new HoloLens 2 AR headset. But the high end offerings are costly and the low end ones are bulky and relatively heavy, adding weight at the worst possible area of a headset (directly in front).
Delivering high quality finger tracking on regular cameras is an entirely different level of challenge. However, if it can be done it allows finger tracking to be added at no extra hardware cost to headsets which already use cameras for other tracking.
In 2016 at Oculus Connect 3, Chief Scientist Michael Abrash made a range of predictions about VR in the year 2021. He noted that finger tracking could be done perfectly with markers on gloves (and showed this off), but not yet directly. However, he predicted that by 2021, it would be possible:
We’re now just two years away from 2021. This lines up with our expectation that Rift S could one day receive an update to support it.
Machine Learning Breakthrough
Last year at Facebook’s annual F8 conference the company finaly showed off high quality gloveless, markerless finger tracking:
An Oculus representative explained on stage that this “breakthrough” was made possible by Facebook’s heavy investment in machine learning (“AI”) research. They use the kind of glove system described by Abrash in 2016 to train a convolutional neural network to recognize hand poses.
The results look impressive. However, Facebook hasn’t demonstrated this technology to journalists or developers yet. We therefore don’t know how well it works outside a lab environment.
Just over five months later at Oculus Connect 5, Carmack made his statements about finger tracking on Quest, seemingly referencing this project.
Valve’s Headset & Vive Cosmos
Facebook isn’t the only PC VR company with finger tracking ambitions of course. In October last year HTC announced finger tracking is coming to the Vive Pro using its two front facing cameras:
HTC will show off this finger tracking at GDC next month. Of course, the Vive Pro‘s four figure price puts it out of reach for most consumers. But the upcoming Vive Cosmos also has cameras on the front, and it may be significantly more affordable. That opens up the possibility that Cosmos could also have finger tracking.
In November last year, images leaked online of a prototype Valve VR headset. Sources close to UploadVR confirmed this was an in development first party headset. Not only did the headset have two front facing cameras, but it also had a slot remarkably similar in shape and configuration to the Leap Motion finger tracking addon.
Given that startup VR headset company Pimax offer a $169 Leap Motion addon for their 8K and 5K+ headsets, it could be possible that Valve plans the same. Or perhaps Leap Motion was simply being used to help train a first party algorithm using the front facing cameras. If the Valve headset releases this year it could be direct competition for Rift S, so it will be interesting to see what features it offers.
The Potential
When people put on a VR headset, one of the first things they do is reach out with their hands. Tracked controllers offer a basic representation of our hands. They’re also very suited for gaming. But they don’t track the vast majority of finger motion and the very act of holding them restricts those motions too.
In non-interactive VR experiences and social VR, controllers often feel more like a chore than a help. If we could enter these experiences by just putting on a headset and seeing our real hands, this reduction of friction would be a welcome improvement.
Enterprise customers using VR for tasks like architecture visualization often forgo controllers entirely due to this friction. Gaming-style controllers are simply much less appealing to non-gamers than directly seeing your hands.
In social VR the ability to gesticulate freely with your real hands adds to immersion and increases social presence. I’ve spent a lot of time in platforms like AltSpaceVR- Leap Motion is by far my preferred method of input.
Given Facebook’s tone when discussing the research it seems significantly unlikely that Rift S could have finger tracking at launch, or even soon after. But given Carmack’s comments, when the software becomes ready and if Facebook wants to, it could bring finger tracking to Rift S one day.
