Meta today showed off an AI powered ‘concept’ for creating VR worlds with your voice, called Builder Bot.
Instead of picking objects from a user interface and placing with controllers, Builder Bot lets you create by simply describing out loud what you want. In the demo, Mark Zuckerberg asks for a beach, then for a specific type of clouds, then props like trees and a picnic. His colleague is even able to ask for specific ambient background sounds and Zuckerberg asks the stereo to play a music genre.
Zuckerberg describes Builder Bot as a “concept”, warning that “there are a lot of challenges we still need to solve”. The delay between giving a command and seeing it actualized in the demo seems impossibly short. Further, Meta didn’t make clear how much of the demo is actually real, nor whether the 3D models are dynamically generated or picked by the AI from a library.
Meta says Builder Bot is possible thanks to self-supervised learning (SSL), a relatively new way to train AI models the company has helped pioneer in recent years. Most AI today uses supervised learning (SL) requiring vast amounts of data carefully labeled by humans. But that’s obviously not how humans or animals learn, and Meta’s researchers say relying on labelling is a bottleneck to developing more generalized AI. With SSL, AI can get a deeper understanding of a concept from far less data that doesn’t need to be labeled. Existing projects like OpenAI’s DALL·E use SSL to generate images you describe with text, but this is the first time we’ve seen this idea applied to virtual world creation.
With Horizon Worlds (which this demo seems to be based on) Meta is already trying to lower the barriers to creating VR experiences. Like Rec Room, Worlds lets you build inside VR by using your controllers to place & manipulate shapes and using a visual scripting system to add dynamic functionality. But not everyone is comfortable using console-like controllers to navigate intricate menu systems. Projects like Builder Bot hint at a future where just like in Star Trek, anyone can create their own virtual worlds with just their voice.
