An image shared on Twitter of the upcoming Magic Leap 2 shows cameras on the controllers, used for inside-out tracking.
In January we reported on Magic Leap 2’s specs – or at least some of them – being shared at SPIE Photonics West 2022. But as it turns out, we missed that the company also revealed that the controller itself uses inside-out tracking. It wasn’t clear at the time what exactly that meant, but the image shared this week by entrepreneur Peter Diamandis shows two forward-facing cameras.
Controllers in almost all AR and VR systems available today are either tracked by the headset or rely on external base stations. Meta’s Quest 2 for example tracks a pattern of infrared LEDs underneath the plastic ring of its controllers, while Valve’s Index controllers determine their position relative to SteamVR “Lighthouse” base stations placed in the corner of your room.
Relying on the headset for tracking has a flaw: if the controller moves out of view of the sensors or if any part of your body gets in the way, tracking will temporarily break. This isn’t a problem for many use cases, but does limit intricate two handed interactions and scenarios like looking left while shooting right. Using external base stations can alleviate most of these issues, but that increases setup time and severely limits portability – and the path from controllers to base stations can still be occluded.
Magic Leap 1 and Pico Neo 2 used magnetic tracking. Unlike visible light, the magnetic field can pass through the human body so occlusion isn’t an issue. But magnetic tracking isn’t as precise as optical tracking systems can be, and adds significant weight and cost to the hardware.
Controllers with onboard cameras promise to solve the occlusion problem while maintaining high precision by tracking themselves in the same way inside-out headsets do – using a type of algorithm called Simultaneous Location And Mapping (SLAM). SLAM essentially works by comparing the acceleration (from an accelerometer) and rotation (from a gyroscope) to how high contrast features in your room are moving relative to the cameras. Initial SLAM algorithms were hand crafted, but most today use at least some machine learning.
The potential downsides of this approach are the cost of a chip powerful enough to run the tracking algorithm, the reduction in battery life due to the power that chip would draw, and the need to have a well-lit environment with high contrast features such as posters – though that limitation applies to inside-out headsets too. Some have suggested tracking quality may be reduced in fast movements due to motion blur, but this shouldn’t be any more of an issue than tracking fast moving LEDs – a global shutter sensor with a low exposure time should make this a non-issue.
Meta is seemingly also planning to use controllers with onboard inside-out tracking in its upcoming Project Cambria headset. Images of Quest-like controllers with cameras instead of an LED ring first leaked in September, and the rings aren’t present in the official reveal trailer either.
Both Magic Leap 2 and Project Cambria are slated to release this year, though neither has a specific release window. They’re very different products – ML2 is a transparent AR headset designed for enterprise while Cambria is an opaque headset for VR and mixed reality – but whichever launches first will be the first AR or VR system to use this new approach to controller tracking.
