Meta and BMW teamed up to get a Quest Pro's tracking working in a moving vehicle.
If you try to use inside-out tracking in a car today, in a headset like a Quest or even on a phone, you'll notice the virtual world (VR) or virtual objects (AR/MR) drifting through the cabin as the car turns and accelerates - the content won't be stable.
That's because these systems use both the inertial measurement units (IMU) and cameras to determine your position. While the cameras see the car's interior as stationary, the accelerometer and gyroscope in the IMU sense the car's acceleration and steering. This conflict is the cause of the drifting and shifting.
The two companies first announced a partnership on this problem in late 2021, with the stated goal of exploring how AR glasses could be eventually integrated into cars, for both passenger entertainment and as a driver aid.
Today Meta researchers presented the progress made so far. They began by connecting the car's IMU to Project Aria research glasses to feed through acceleration & steering data in real-time, so the tracking system can filter it out. The Aria glasses don't have any form of display though, so they transferred this approach to a Quest Pro - Meta's first headset with color mixed reality - to test out actually anchoring virtual objects to the interior of the car.
Meta claims the virtual content stays stable even when the car is making rapid turns, accelerating quickly, or going over road bumps.
The researchers say the next step will be to access the car's precise 6DoF positioning system to enable seeing world-locked virtual objects outside the windows, such as markers on landmarks and other points of interest.
Meta and BMW aren't the first companies to explore in-car headset usage. In fact, in Germany owners of some Audi cars can already buy a HTC Vive Flow based in-car VR entertainment system called Holoride, which we tried out earlier this year. But Holoride avoids the drifting problem by altogether by disabling the headset's positional tracking and using custom content that uses the car's real steering, accelerating, and braking as virtual movement. That means it wouldn't work for cabin-locked mixed reality, and you can't lean.
BMW's partnership with Meta is currently solely in the realm of research, not product development. There was no announcement of any current or future BMW car actually supporting Quest headsets.
“It is too early to tell exactly how or when this technology will make it into customers’ hands", said Claus Dorrer, Head of BMW Group Technology Office USA. "The research partnership with Meta will allow us to discover what immersive, in-vehicle XR experiences could look like in the future and spearhead the seamless integration of such devices into cars."
In the long term, Meta sees research like this having the potential to "revolutionize travel in cars, trains, planes, and beyond, unlocking new forms of hands-free communication, entertainment, and utility — giving us far more value than the screens and instruments we’re used to seeing in vehicles today".
