"We have a camera passthrough API coming early next year."
In a surprise announcement at Connect 2024 today, Meta revealed that it will give developers access to the passthrough cameras of Quest headsets via a new API next year.
"This will enable all kinds of cutting-edge MR experiences. You've got tracked objects, AI applications, fancy overlays, scene understanding, and so much more."
While mixed reality headsets like Meta Quest 3 and Apple Vision Pro use cameras to let you see the real world, today only the system and certain first-party apps actually get raw access to the cameras. Third-party developers can use camera passthrough as a background, but their apps don't get to actually see this passthrough. They instead get higher-level data, such as hand and body skeletal coordinates, a 3D mesh of your environment with bounding boxes for furniture, and limited object tracking capabilities. That means they can't run their own computer vision models, which severely limits the augmentation capabilities of these headsets.
David Heaney
With visionOS 2, Apple is now giving enterprise companies raw access to Vision Pro's passthrough cameras for non-public internal apps, but this requires a special licence from Apple and is restricted to "in a business setting only".
Meta hasn't yet said whether there'll be any restrictions to the Passthrough API, but we'll keep a close eye on the company for any further details of this exciting upcoming capability, which could unlock entirely new use cases for Quest headsets.
