Meta is working on making Quest headsets work in moving vehicles.
UX Designer Andrew Fox reposted a video to X of Facebook user Netsvietaiev Oleksandr trying to use Quest 3 in passthrough mode on an airplane, but being unable to because the interface drifts away.
Meta's CTO Andrew Bosworth replied to the post to summarize why it was happening, and finished by saying Meta is "working on it".
No, we use an IMU to keep objects localized relative to your headset motion so moving vehicles represent a challenge (when they accelerate in any direction). We also use the cameras, of course, they work together as the IMU is higher frequency but lower accuracy. Working on it!
— Boz (@boztank) January 24, 2024
So why does this drifting in moving vehicles actually happen?
People often think that markerless inside-out tracking systems on headsets, glasses, and self-tracking controllers only use the cameras, but this isn't the case. These systems rely just as much on the inertial measurement unit (IMU), a chip that contains a tiny accelerometer and gyroscope.
While cameras typically run at 30Hz or 60Hz, the IMU typically provides updates around 1000Hz, enabling much lower latency. However, an IMU can't actually detect absolute movement - at least not directly. The accelerometer in the IMU senses acceleration relative to gravity though, and you can take the integral of acceleration over time to get velocity. And if you then take the integral of those velocity values over time, you get displacement from the original position.
This process is called dead reckoning. From moment to moment, it’s how every headset and controller tracks itself, and the optical component like cameras or laser base stations are only used to correct for the cumulative error caused by the noisyness of the IMU data. But in a moving vehicle such as an airplane, the accelerometer will pick up the acceleration of the vehicle itself, thinking it's headset acceleration, which results in the sudden and rapid positional drifting you see in Fox's video above.
Apple Vision Pro has a Travel Mode, but it's unclear what exactly it does.
It's unclear what exactly Bosworth means by saying Meta is "working on" the problem.
Currently it's possible to disable 6DoF positional tracking on Quest, and the headset will fall back to 3DoF rotation-only tracking, but that also disables passthrough. So Meta could perhaps be planning to allow passthrough when positional tracking is disabled.
Alternatively, Meta might be working to somehow make positional tracking work in moving vehicles, perhaps by instructing the system to rely entirely on camera visuals and ignoring the IMU. Last year Meta demonstrated a research project making a Quest Pro work in a moving BMW, but this involved interfacing with the car's own onboard IMU.
Meanwhile, Apple Vision Pro has Travel Mode for use on airplanes, and Apple has marketed this use case, but it's currently unknown what exactly Apple's Travel Mode does either. Apple has only said it will "stabilize visuals". I plan to test this feature on my journey home after picking up Vision Pro in New York.
