Meta CTO Andrew Bosworth gave his thoughts on the idea of giving Quest developers access to the passthrough cameras.
While mixed reality headsets like Meta Quest 3 and Apple Vision Pro use cameras to let you see the real world, only the system and certain first-party apps actually get raw access to the cameras. Third-party developers can use camera passthrough as a background, but their apps don't get to actually see this passthrough. They instead get higher-level data, such as hand and body skeletal coordinates, a 3D mesh of your environment with bounding boxes for furniture, and limited object tracking capabilities. That means they can't run their own computer vision models, which severely limits the augmentation capabilities of these headsets.
With visionOS 2, Apple is now giving enterprise companies raw access to Vision Pro's passthrough cameras for non-public internal apps. While this requires a special licence from Apple and is restricted to "in a business setting only", the move has renewed questions to Meta as to whether it plans the same.
In a wide-ranging interview with Matthew Ball, the author of the book The Metaverse: And How It Will Revolutionize Everything, Meta CTO Andrew Bosworth was asked about the issue and gave his full extended thoughts:
"The serious answer is we can all imagine phenomenally useful use cases if a developer has direct access to the camera. You're a mechanic and you're looking at an engine that you're not familiar with and a developer could build the types of tools to help you see the overlays in the schematic, even diagnose the problem.
Meta is never going to build that, and the developer can't upload to us every possible configuration of image that we might see for us to build a classifier on, that's not credible either. That's a use case that goes underserved if you don't build out the capability.
At the same time, when this technology is so new in the world, we also do want to make sure that bystanders feel comfortable if someone chooses to wear the headset, for example, on an airplane, that they understand what the implications for them are. Now we're finding more and more techniques to address this. There's different ways you can do bystander signaling.
I also think the consumer markets are more and more familiar with the technology. I made a little joke about Apple earlier - let me turn it around and say I think the Apple Vision Pro is great for our whole industry. There's really broad understanding of these devices now, and the more they're understood, the less fear there is, I think, that a bystander is going to feel fear or that you're going to feel bad putting it on for fear that somebody else would have a negative reaction to it, or they'd be surprised by it.
So I think we do always think of technology, in our industry, as almost in its own terms. That's not fair. Technology exists inside the context of a society. And the more comfortable society is with the technology, the more you can kind of execute freely and the less comfortable the society is, the more cautious you need to be as you bring that technology to market, because if you're not careful you could actually impede its adoption over time.
So I think we took a pretty conservative stance there. I think everyone probably understands why we took that conservative stance - I stand by that as the right thing to do at the time. But of course, at the same time, we are thrilled with the potential applications that mixed reality has. If a developer with full consent from the consumer and an understanding audience contextually around the person using is ... if we can unlock that functionality, I think we're all excited about what that represents.
So yeah, we'll keep looking at it: how we see consumer comfort, let's call it, with this technology evolving, and what kind of power that unlocks."
We highly recommend listening to the full interview, which is available as an episode of Bosworth's podcast Boz To The Future on platforms like Spotify and Apple Podcasts.
David Heaney
In it, Bosworth is also asked about topics such as Meta's enormous spending on AR/VR, what it will take to sell 100 million units of a headset, reverse passthrough (EyeSight), the path to transparent AR glasses, the role of AI for AR/VR, and the specifications he expects from a Quest headset in 2031.
