Meta shared new research toward realistic simulation of clothes for avatars.
This clothing solution is built on top of Codec Avatars – Meta’s long term project to develop photorealistic avatars driven in real time by VR hardware sensors – and comes from the same team. Instead of traditional rendering, the idea of Codec Avatars is to use a series of neural networks to learn the appearance of a given person, then constantly encode their current state based on sensor input, and finally decode this state as geometry and final output textures. Since originally presented, the Codec Avatars team has showed off several evolutions of the system, such as more realistic eyes, a version only requiring eye tracking and microphone input, a 2.0 version that approaches complete realism, and generation using only an iPhone with FaceID.
The latest research, Deep Photorealistic Appearance for Physically Simulated Clothing, applies neural networks to realistic deformation of clothing across any body type:
As we always note, Meta still has a long way to go to reach this kind of fidelity in shipping products. This clothing system currently runs at just 13 frames per second on an NVIDIA RTX 3090 GPU. But in the field of machine learning optimization can be dramatic over time. The speech synthesis and object recognition algorithms on smartphones today also once required expensive PC GPUs.
Meta Avatars today have a basic cartoony art style. Their realism has actually decreased over time, likely to better suit larger groups with complex environments in apps like Horizon Worlds on Quest 2’s mobile processor. Codec Avatars may, however, end up as a separate option, rather than a direct update to the cartoon avatars of today. In his interview with Lex Fridman, CEO Mark Zuckerberg described a future where you might use an “expressionist” avatar in casual games and a “realistic” avatar in work meetings.
In April Yaser Sheikh, who leads the Codec Avatars team, said it’s impossible to predict how far away it is from actually shipping. He did say that when the project started it was “ten miracles away” and he now believes it’s “five miracles away”.
