Facebook CEO Mark Zuckerberg wants eye tracking & facial expression tracking in future iterations of Oculus Quest.
In a wide-ranging interview with The Information, Zuckerberg talked about the company’s strategy & direction in the AR/VR space.
He said Facebook is already working on its next VR headsets:
“Because of how hardware gets developed, you kind of need to know what your next three products are going to look like all at the same time. It’s not like software where we’re changing it every couple of weeks. We have product teams spun up now working on the next few generations of virtual reality and what Quest 3 and 4 are gonna look like.“
Zuckerberg revealed one of his personal most-wanted features in upcoming Oculus headsets; eye & face tracking to drive avatars in social experiences:
“One of the things I’m really excited about for future versions is getting eye tracking and face tracking in, because if you’re really excited about social presence you want to make sure the device has all the sensors to really kind of animate realistic avatars so you can communicate well.“
Later in the interview he spoke about some of the difficulties in making this a reality:
“On the VR side, I think the biggest things that we’re very focused on now are: how do you pack more sensors, to create a better social experience, into the device? To run each sensor requires more CPU power and that generates more heat and creates all these different issues.
When I think about where you’re at with VR today, there’s some pretty great games & different experiences but I’d love to get to the point where you have realistic avatars of yourself, and where you can make real authentic eye contact with someone and have real expressions that get reflected on your avatar.
So what do you need for that? Well you need to be able to do eye tracking to do eye contact. You need to have some sense of face tracking or sense of what’s going on with the person’s expressions in order to be able to have those emotions come through naturally.“
VR headsets today with eye tracking or face tracking are geared toward enterprise buyers. Pico’s Neo 2 Eye is the first standalone headset with eye tracking, priced at $900 with a focus on enterprise use cases. HP’s PC-tethered Reverb G2 Omnicept Edition will have eye tracking & face tracking, but is also focused on enterprise use cases with pricing not yet revealed.
Facebook’s Avatar Research
Facebook first showed off its long term research on photoreal ‘Codec Avatars’ back in March 2019.
Powered by machine learning, the avatars are generated using a specialized capture rig with 132 cameras. Once generated, they can be animated in real time by a prototype VR headset with eye tracking & face tracking cameras.
When first presenting codec avatars, Facebook warned the technology was still “years away” for consumer products – shipping this kind of photoreal avatar will require a number of breakthroughs.
Facebook’s Avatars Today
Oculus Quest today has a built-in basic avatar system, Oculus Avatars. It’s used in a few apps like Poker Stars VR and Tribe XR DJ, but not much else. Platforms like Bigscreen, VRChat, and Rec Room use their own separate avatar systems.
Current headsets don’t have eye or face tracking, but Oculus Avatars uses a neural network to simulate lip movement, and developers can set up priority-ranked gaze targets to simulate eye movement.
Back in September, Facebook announced the new Facebook Avatars system will replace Oculus Avatars. It’s apparently an evolution of the current Facebook VR avatars used in Facebook Horizon and the beta for the new Venues. Staffers working on the project include former Pixar animators.
Facebook Avatars seems to take a step backward in artistic realism, but adds a complete torso and simulated arms.
In today’s interview Zuckerberg announced Facebook Avatars will ship this year, saying it will “get more and more realistic over time”, suggesting the company intends to take an iterative approach. Zuckerberg also commented on Facebook’s low cost hardware strategy.
