Meta’s researchers are approaching complete realism with Codec Avatars 2.0, their prototype VR avatars using advanced machine learning techniques.
Facebook first showed off work on ‘Codec Avatars’ back in March 2019. Powered by multiple neural networks, the avatars are generated using a specialized capture rig with 171 cameras. Once generated, they can be driven in real time by a prototype VR headset with five cameras; two internal viewing each eye and three external viewing the lower face. Since then the researchers have showed off several evolutions of the system, such as more realistic eyes and a version only requiring eye tracking and microphone input.
At MIT’s Virtual Beings & Being Virtual workshop in April Yaser Sheikh, who leads the Codec Avatars team, showed a video of the latest version of the project, described as “Codec Avatars 2.0”:
“I would say a grand challenge of the next decade is to see if we can enable remote interactions that are indistinguishable from in person interactions“, Sheikh remarked.
In a paper published last year, Sheikh and his colleagues claim their newest models are smaller and more efficient than their past research, with the neural network now computing only the pixels visible to the headset. With this advancement, a Quest 2 headset is apparently able to render five avatars in an otherwise empty scene at 50 frames per second.
Still, the company seems to have a long way to go to reach this kind of fidelity in shipping products. Meta Avatars today have a basic cartoony art style. Their realism has actually decreased over time, likely to better suit larger groups with complex environments in apps like Horizon Worlds on Quest 2’s mobile processor.
Codec Avatars may however end up as a separate option, rather than a direct update to the cartoon avatars of today. In his interview with Lex Fridman, CEO Mark Zuckerberg described a future where you might use an “expressionist” avatar in casual games and a “realistic” avatar in work meetings.
During the workshop Sheikh noted that it’s impossible to predict how far away Codec Avatars is from actually shipping. He did however say that when the project started it was “ten miracles away”, and he now believes it’s “five miracles away”.
