Meta today showed off an AI powered ‘concept’ for creating virtual reality worlds with your voice, called Builder Bot.
Instead of placing objects from a user interface with controllers, Builder Bot is shown to let you create by simply describing out loud what you want. In the demo, Mark Zuckerberg asks for a beach, then for a specific type of clouds, then props like trees and a picnic. His colleague is even able to ask for specific ambient background sounds and Zuckerberg asks the stereo to play a music genre.
Zuckerberg describes Builder Bot as a “concept”, warning that “there are a lot of challenges we still need to solve”. The delay between giving a command and seeing it actualized in the demo seems impossibly short. Further, Zuckerberg didn’t say how much of the video is actually real software, nor whether the 3D models are dynamically generated or picked by the AI from a library.
Meta says ideas like Builder Bot are becoming possible thanks to self-supervised learning (SSL), a relatively new way to train AI models the company and others like Google and OpenAI have helped pioneer in recent years. Most AI today uses supervised learning (SL) requiring vast amounts of data carefully labeled by humans. But that’s clearly not how humans or other animals learn, and Meta’s researchers suggest relying on labelling is a bottleneck to developing more generalized AI. SSL results in AI with a deeper understanding of a concept using less data that doesn’t need to be human labeled. OpenAI’s Codex can create simple 2D games from text requests, but this is the first time we’ve seen this idea applied to VR world creation.
With Horizon Worlds, Meta is already trying to lower the barriers to creating VR experiences. Like Rec Room and , Worlds lets you build inside VR by using your controllers to place & manipulate shapes and using a visual scripting system to add dynamic functionality. But not everyone is comfortable using console-like controllers to navigate intricate menu systems. Concepts like Builder Bot hint at a potential future where just like in Star Trek, anyone can create their own virtual worlds with just their voice.
