Quest 2 apps can now use your walls & furniture from Room Setup.
Room Setup shipped last week as an experimental feature. It lets you mark out your walls, doors, windows, and furniture from inside Quest 2’s real world passthrough view, using the VR controllers.
With the release of v40 of the Quest SDK developers can now access these walls, doors, windows – as well as your couch and desk – to build mixed reality applications. Marking your couch was added to the Quest OS in February 2021, and marking your desk in April 2021, but developers haven’t been able to access these until now.
An example app from Meta called The World Beyond is available on App Lab, and the assets and source code are available on GitHub.
Quest 2’s passthrough view is low resolution and black & white – it was originally only intended for setting up your Guardian boundary. So while apps using this new scene understanding will run on Quest 2, it’s clear the real purpose of the new functionality is for Project Cambria, Meta’s upcoming “high end” headset with high resolution color passthrough. From a practical perspective, Quest 2 seems to be an affordable development kit for mixed reality, not the ideal target device.
Having to mark out walls, doors, windows, couches, desks and furniture manually is a fairly arduous and imprecise process. Last week Meta’s CTO explained why this isn’t yet automatic, saying “Segmentation is getting better all the time but still has error. The risk of getting it wrong is a concern as it relates to how people can safely navigate a physical space.” Consulting CTO John Carmack went into more detail on Twitter, saying “There are a good number of smart people at Meta working on understanding the world from camera images, but most of it isn’t production ready.”
But will this always be the case for Project Cambria? It will feature much higher resolution cameras and active depth sensing, which Meta says “helps reconstruct surfaces with higher fidelity and accuracy”. In the announcement of the v40 SDK, Meta Product Manager Rangaprabhu Parthasarathy hinted “in the future, devs will be able to add more bells and whistles to their apps that are only available on this device”.
Documentation for Scene Understanding in Unity is available on the Oculus Developer website.
