Meta recently showed press its first in-person look at progress toward a wristband that can sense your intentions.
In 2019, the company-formerly-known-as-Facebook acquired a startup called CTRL-Labs that was working on a wrist-worn device for reading the electrical activity heading toward your hand. It does this with EMG, or electromyography, which reads the activity with sensors strapped around the circumference of your arm.
Before CTRL-Labs was swiped up by Mark Zuckerberg only a few news outlets tried its prototype, among them The Verge’s Adi Robertson writing “I tried the wristband that lets you control computers with your brain.”
We still haven’t worn the wristband yet — it requires significant training time in its current state — but Zuckerberg and those he employs believe Thomas Reardon’s team from CTRL-Labs are onto something that could be the key to unlock for the 21st century what the QWERTY keyboard layout did in the 19th century and the graphical user interface (GUI) did in the 20th century.
They’ve been searching for an input system that makes sense to pair with Augmented Reality glasses and, in the tech acquired from CTRL-Labs, they believe they’ve found it.
Why A Wristband?
Voice input only gets you so far because there’s plenty of places where you’d prefer to be silent than audibly say what you’re thinking. Large gestures recognized by hand tracking are a no-go for the same reason. You just can’t do it everywhere without looking like a fool. Touch controllers, meanwhile, don’t make sense to be carried with you everywhere you go.
The idea is this wristband, watch, bracelet — whatever it ends up getting marketed as in a consumer product — could pair with future eyewear and “co-adapt” with its wearer to understand and translate intent speedier than today’s phones and personal computers.
“The two primary advantages of this are….how responsive it can initially be,” Zuckerberg responded to a question from UploadVR. “If you wanna start typing something on your phone, you have to go through this whole activation of take out your phone, unlock it…and then you can get going. And on this, it’s gonna be extremely quick. So the activation is one.”
Zuckerberg wore the wristband and demonstrated how tiny gestures — even those practically imperceptible to an onlooker — could translate to the sorts of simple actions we do hundreds of times a day for things like moving through a TV’s menu via remote control or swiping from one screen to the next on your phone.
“Then…what is your steady state bandwidth of communication? And this,” he says, referring to the bulky device on his wrist, “I think will initially be superior on activation while having a relatively lower bandwidth of communication for very high complexity things. But then eventually…say within the next five or six years…that’s not an unreasonable timeframe to say that this is really the highest bandwidth form of communication that you have too.”
Friction & Focus
As Zuckerberg put it, Quest VR headsets and Horizon with its avatar-populated 3D realms are two of what he sees as four interrelated platforms he’s prioritizing for the future of Reality Labs. The remaining two are still clearly in the rawest research and product development phases: AR glasses and neural interfaces like the wristband.
“We’re gonna really focus on those four areas and probably deprioritize other stuff in Reality Labs,” Zuckerberg said. “There’s interplay between them obviously, but they also I think each will stand on their own. By the time that you have neural interfaces there’ll still be many more people using phones and computers than using VR or AR devices, even if it’s initially invented in order to be a kind of input mechanism for augmented reality.”
If you’ve been a VR headset owner in the last decade you’ll be familiar with this “activation” issue. John Carmack, a leading technical guide for Oculus-then-Facebook-then-Meta, has called the problem facing headsets “friction”. That is, everything from keeping your battery charged to entirely blocking out the physical world to the seconds-long wait at startup robs the wearer of precious moments better spent doing something else.
Some point to this friction as a reason “VR is dead” and Meta misspent tens of billions of dollars it made from targeting ads against the personal details of much of the planet. If there’s a more useful narrative to pay attention to than this chorus, it’s that Meta’s teams are obsessed with paying attention to data and using it to slip people further into spaces they control for longer periods of time. In other words, they’re experts at reducing friction.
Examples:
– Quest Pro is more comfortable than any headset Meta has shipped before while keeping its wearers aware of their physical surroundings as often as they want.
– Horizon Worlds erases the need for big downloads before visiting 3D virtual worlds accessed from the Internet.
– Horizon Home opens up Quest headsets to visitors from the moment the headset is unlocked.
If you put the neural wristband effort in this larger context and if it can truly be faster than unlocking your phone then you can start to understand the end goal of Zuckerberg’s metaverse effort.
Faster Than The QWERTY Keyboard?
The biggest claim of the day about the impact of this work came from Meta’s CEO himself. As Zuckerberg said, “right now the bit rate is below what you would get for typing quickly”.
Meta’s live demos of the wristband seemed imperfect even when worn by Zuckerberg and his CTRL-Labs researchers. Still, there are more than just hints that the very top of Zuckerberg’s leadership team see a much bigger picture.
In trying to contextualize the weight of this work, Meta CTO Andrew Bosworth cited the invention of the QWERTY keyboard layout while the company’s top researcher, Michael Abrash, pointed to Doug Engelbart and the invention of the graphical user interface. Meta’s leaders appear to be drilling down into the very foundations of modern technology’s first principles. Searching, perhaps, for the right entry point and strategy to lay a foundational layer for the next 50 years of personal computing.
“Mark is alluding to this idea of bit rate,” Reardon said. “And we do really think it’s almost impossible for it to not get faster. This is just a consequence of the way your motor nervous system works, that you rapidly adapt to the task and if you get the proper feedback, you just get faster at it. You can’t help but get faster at it as you minimize the amount of movement that you do…I don’t want to talk about products here, but I would just say we have really, really compelling research results towards that outcome.”
When pressed on the question of exactly what use cases the input system is ready to replace in its current state, Bosworth responded “Reardon could replace his keyboard.”
“We plan on taking it past any kind of experience you would have today with a mechanical controller,” Reardon said. “Something like the Touch controller — there’s no reason why we can’t go well beyond that.”
“You can imagine we need to figure out what the language and interface is gonna be to interact with this,” he said. “But you have a very large number of gestures and have this complex system where you’re communicating through a combination of these, or we might just make it so you’re just writing things out.”
