A VR developer found a lower friction way to access Quest 3's passthrough cameras.
Meta does not officially allow third-party developers to access the cameras of Quest headsets, but earlier this week we reported on a clever workaround from Michael Gschwandtner the co-founder and lead developer of the hand tracking fitness app XRWorkout.
David Heaney
Gschwandtner's technique involved having you use Quest's casting capability to cast to a WebView inside his app which loaded the oculus.com/casting URL. But this requires painstakingly logging into your Meta account in this WebView, and requires the user to take action by casting.
Another VR developer, Julian Triveri, has now figured out a technique that doesn't involve a WebView or casting, and thus doesn't require logging in or initiating a system action.
Triveri realized that the Android media projection API, designed to enable screen capturing for custom recording or casting setups, works on Quest headsets. On Quest it returns 1024×1024 frames of the central 82 degrees of the field of view, slightly lower than you'd get with casting but still usable to run custom computer vision models.
The API requires a user-granted permission, but after that the app now has full access to what the user is seeing.
Triveri released an open-source demo of this technique on GitHub, and UploadVR has confirmed it works as described. He even added a branch showing the technique being used to drive AprilTag tracking (a type of fiducial marker).
An interesting limitation of this method is that it can't be used while recording or casting on the headset, so Triveri had to use a PC to record footage. Like with Gschwandtner's earlier technique, the other limitation is that the view captured includes virtual objects and interfaces, not just the camera view, and this could interfere with computer vision tasks.
What Happens Next?
With Triveri's discovery, any Quest app can now conceivably essentially access the cameras with minimal friction. Will Meta allow this approach to propagate? Or could the company crack down by patching it in a future release of its Horizon OS?
Last month Meta CTO Bosworth gave his thoughts on the idea of intentionally giving Quest developers access to the passthrough cameras during an interview, saying Meta will "keep looking at" the issue.
David Heaney
In comparison, for its Vision Pro headset Apple only gives raw access to enterprise companies, and only for non-public internal apps.
