The Oculus Quest room scale standalone headset launched this week. In a Facebook post, Facebook’s CTO Mike Schroepfer revealed that the company initially had “vigorous debates” about whether the product would even be possible:
I remember the vigorous debates about whether it was possible to build the product. Inside-out tracking was in the research phase and the idea of doing it on a mobile chip was thought impossible by many. Now the product “just works.”
Schroepfer reports directly to Mark Zuckerberg. Even Facebook’s VP of VR/AR, Andrew Bosworth, reports to Schroepfer, not Zuckerberg directly.
This quote hints at the long development cycle of Quest, a product that was in development for years, hinging on breakthrough technologies that didn’t exist when it was conceived:
Quest Was Always The Goal
Oculus Quest has been in the making for over three years now, but a product like it has been the general goal of Oculus for much longer.
In a video interview with Kotaku in 2012 (before even formally joining Oculus) now CTO John Carmack described a headset that would use “mobile phone hardware” and cameras for positional tracking, with no wires. In a 2013 interview with Endaget, now at Oculus, Carmack even described this ideal headset as running a version of Android.
When the Rift and Touch hardware were finalized in 2015, Oculus began an ambitious project. The goal was to bring the same Rift+Touch experience, minus some graphical fidelity, to a completely standalone headset. The project would come to be called Project Santa Cruz. As Schroepfer stated this week and as Sean Liu stated at Oculus Connect 4, many at Facebook thought it was impossible.
Inside-Out Tracking: A Research Problem
One of the greatest challenges of Project Santa Cruz was getting high precision inside-out tracking for a headset and two controllers running on mobile hardware. In fact, getting it running at all was an unsolved problem to begin with.
While today there are other standalone headsets with inside-out head tracking, none exist (other than Quest) that also use the cameras for controller tracking. The HTC Vive Focus Plus features 6DoF controllers, but they are tracked with ultrasonic hardware, not computer vision software. Like the Lenovo Mirage Solo, the cameras on the Focus Plus are only used for the headset itself.
Microsoft introduced inside-out tracking to the VR market with Windows MR in late 2017, but this relies on the processing power of a PC.
Back when Project Santa Cruz began high quality inside-out tracking was considered an area of research, not shippable technology. To build up a team to solve it, Oculus acquired a number of computer vision startups, including 13th Lab and Surreal Vision.
Inside-Out Tracking: Possible On Mobile?
Just before Oculus Connect 2 in 2015, Oculus founder Palmer Luckey stated “VR-grade inside-out tracking is not currently workable on mobile devices.”
During the conference, Carmack elaborated on the difficulty of this problem, and criticized the company’s research focus over optimizing the core technology for mobile hardware:
“It does not look good for making an inside out tracking system that doesn’t consume a whole lot of battery power […] We have like 30 computer vision experts at Oculus from the different companies we’ve acquired and none of them want to just go solve this problem. They’re all working on their esoteric, kind of researchy things while this is a problem that I want solved right now. I wish somebody had spent all of this last year on it.”
Getting tracking running on a standalone headset was useless if it drained the battery in minutes. But Carmack and others at Oculus continued work, and by February 2016 the CTO was spending “most of” his time on position tracking.
Hardware Acceleration, Active Cooling
At that time in early 2016, Carmack’s purpose for inside-out tracking seemed to be to bring it to the Gear VR, the phone based VR solution Oculus had partnered with Samsung to create. New hypothetical future products were one thing, but the Gear VR experience could be radically improved with positional tracking.
Unfortunately, positional tracking never came to Gear VR (at least not officially), primarily due to the kinds of issues Carmack highlighted at Oculus Connect 2. The other issue however was that of cooling.
The Gear VR’s main flaw was that it overheated frequently during gameplay, forcing a system shutdown. This could happen as quickly as 15 minutes depending on how much the game strained the system. Running a VR game, the cameras, and advanced computer vision algorithms at once would make this problem even worse
Quest uses two methods to overcome these issues.
Thanks to a close partnership with Qualcomm established for the Gear VR and later the Oculus Go, Oculus was able to code the tracking to work on the Hexagon DSP rather than the CPU. DSPs are dedicated programmable chips designed specifically for sensor and image processing. Smartphones mostly use the DSP to enhance the photos taken by their cameras- Quest uses it for VR tracking. Oculus confirmed that positional tracking “doesn’t affect” performance for VR apps.
To solve the issue of overheating, Quest uses an active cooling fan. This has rarely ever been done with ARM processors. The Apple TV 4K, OUYA micro console, and the HTC Vive Focus are the only instances on the consumer market we know of (and the latter is technically only for enterprise).
Monarch
While the mobile tracking was being solved, others at Oculus were designing the hardware. The first challenge was figuring out what hardware would be needed. How many cameras, and at what position? One of the first functional prototypes was called Monarch, referenced in a talk at OC5. Monarch was a proof of concept using large computer vision cameras, two facing forwards and two to the sides.
Next, the team worked on miniaturizing this and finding the optimal camera placement. This resulted in the configuration seen today on the Quest. The decision to recess the cameras was taken so that they would be able to survive regular wear and tear, including drops, without breaking.
Oculus Connect 3: Headset Press Demo
At Oculus Connect 3 in late 2016, Oculus revealed the existence of Project Santa Cruz and gave members of the press a demo of the headset.
The front of the headset was essentially just a Rift with the cameras added. The battery, processor and all other mobile hardware were in the rear behind the head. It’s unclear why exactly Oculus made the decision to put the hardware in the front, but it’s likely due to cost and the inability to lean back seated.
We went hands on with this demo at OC3, and noted:
Tracking did have one small hiccup when I reached a corner of the room and turned quickly, but otherwise tracking was really solid and I felt no discomfort quickly taking several steps across the room. Overall, the experience felt more like a Rift that went wireless rather than a Gear VR that gained position tracking.
Oculus Connect 4: Controllers Online
At Oculus Connect 4 in late 2017, Oculus revealed that the last major development milestone in Project Santa Cruz had been crossed- it now had positionally tracked controllers. We again went hands on with this new hardware at the conference and were impressed by the tracking.
The controllers resembled the Rift’s Touch controllers, but with the tracking ring facing upwards instead of down. Just like the original Touch they are tracked with IR LEDs under the tracking ring, but from cameras on the headset rather than external sensors.
Strangely however, Oculus had opted to replace the buttons and thumbstick with a trackpad. This decision drew criticism from both developers and fans. The thumbsticks were one of the favorite features of Touch, and developers had already built their control schemes around it. Had this decision stuck, they would have to make separate control schemes for Rift and Santa Cruz.
At GDC 2018, Oculus announced that this decision had been reversed. Santa Cruz would now have the same input elements as Rift.
Launch
Seven years after John Carmack first outlined the goal, and almost three years since the headset without controllers was first shown, Oculus Quest is now a $399 product you can purchase at a Best Buy or Amazon.
The development effort to build this product has created a new category of VR device, redefining the VR market that was previously between uninteractive mobile and interactive PC. How well Quest sells is yet to be seen, but it seems unquestionable that standalone will be a major part of VR’s future.
