visionOS 2 adds the ability to track 3D objects that app developers prespecify.
As well as improving hand tracking and enhancing scene understanding, visionOS 2 introduces 3D object tracking, an entirely new tracking capability.
These tracking improvements weren't announced in the main WWDC24 keynote, but Apple engineers detailed them in developer sessions available online.
David Heaney
Since launch Vision Pro has been able to track specified 2D images, such as markers and game boards. With visionOS 2 it can track 3D objects too.
The developer must provide tracked 3D objects as USDZ 3D models to the Create ML tool on their Mac before app compilation. Create ML will train a neural network for each object, which will be baked into the app.
All three objects on the table are real physical objects, not virtual.
That means users can't provide their own arbitrary 3D objects, and tracking new objects would require an app update. Still, this could be a very useful feature for business and education use cases, or even for exhibitions like museums.
visionOS provides a bounding box with scale and orientation for each tracked object, which developers can use to anchor virtual objects to them or even to augment them with 3D effects.
Tracked objects can be augmented.
We'll be very curious to see what visionOS developers end up building with this new capability, which starts to enable true augmentation of specific objects in your space, not just the environment.
The first visionOS 2 beta is available to install already, and the stable release is set for fall later this year.
