Facebook shared some of its research on wrist-mounted devices for high fidelity finger tracking and haptic feedback.
In February The Information reported on Facebook’s formerly-secret plans to release a smartwatch in 2022, with a second-generation device following in 2023. It’s unclear whether this is a separate project. Facebook’s Head of AR/VR Interaction and Input R&D Sean Keller said the following about the potential of a wrist-mounted device:
“Wristbands and watches are already widely used. A device on the wrist can serve as a platform for compute, battery, and radios. It can act as a complete input and output hub with display, touch sensitive surfaces, rich haptics, and can support a broad array of sensors.”
Neural Finger Tracking
Facebook’s Oculus Quest headsets today support controller-free hand tracking, done via the cameras on the headset. It requires a well-lit room, there’s a small but noticeable delay, and it doesn’t work if your hands overlap.
Better cameras, depth sensors, and faster chips could improve this, but there could always be scenarios where the headset can’t see the movement of one or more fingers.
A different approach to hand tracking is to read the neural signals passing through your wrist. Such a system can sense finger movement before it even happens.
Facebook filed a patent application for such a device in early 2019. New York based startup CTRL-Labs had been working on the same idea – Facebook acquired the company in late 2019. What’s being presented now seems to be a refined version of the same technology.
Facebook Neuromotor Interfaces Director Thomas Reardon explained that more of the brain is dedicated to controlling your hands than any other part of the body. He claims a recent breakthrough enables decoding the activity of individual neurons for “almost infinite control over machines”. That means hand interactions could be incredibly precise & reliable – millimeter accurate.
It also means this data can be considered deeply personal, opening up privacy concerns:
“Neural data like this is really quite personal and we treat this issue as part of our research set. The problem of how to deal with information that is this personal and engage in a way that is pro-human and on behalf of users. I will tell you that we are deeply committed to transparency as scientists, to engaging in the world of publishing and the world of public engagement, so that we can explain to people why we use this data, how we use it, and what kinds of user experiences are actually enabled by it.”
Since the device is tracking neural signals, not the actual fingers, you can also simply think about moving your fingers. Reardon showed a short clip of “a person born without a hand” controlling a virtual hand, and outlined exotic use cases such as controlling a virtual hand with six or more fingers.
Wrist-Based Haptics
When talking about Facebook’s long term goals, Head of AR/VR Interaction and Input R&D Sean Keller described pulling out “a pair of soft, lightweight haptic gloves”.
In the shorter term, Research Science Manager Nicholas Colonnese discussed the potential of haptic feedback delivered from a wrist-mounted device. We first heard of Facebook’s work on this when it presented ‘Tasbi’ at the World Haptics Conference 2019.
Colonnese revealed Tasbi stands for Tactile And Squeeze Bracelet Interface and showed off the first footage of it in action:
While intuition would suggest a feeling at the wrist shouldn’t be suitable for finger-based virtual interactions, Colonnese claims “the opposite” is true due to an effect called sensory substitution. Your brain, apparently, interprets the simultaneous visual & haptic stimuli as one event.
“We have tried tons of virtual interactions in both VR and AR with both the vibration and squeeze haptic capabilities of Tasbi. This includes simple things like push, turn, and pull buttons, feeling textures and moving virtual objects in space. We’ve also tried more exotic examples – things like climbing a ladder or using a bow & arrow. We tested to see if the vibration and squeeze feedback of Tasbi could make it feel like you’re naturally interacting with these objects. Amazingly, due to sensory substitution, where your body combines the various multisensory information, the answer can be yes.”
Colonnese also described more mundane use cases for high fidelity wrist-mounted haptics when paired with AR glasses, such as vibrating in the direction of travel for on-foot navigation, or delivering a different feeling vibration for different kinds of inbound communications.
