Collabora using AI to enhance VR hand tracking in Monado, the open-source OpenXR runtime

Virtual Reality and Augmented Reality (XR) is continuing to be a very interesting space with lots of innovation going on and Collabora are at the front of this with Monado, the open source OpenXR runtime.

Reminder: Monado is the first OpenXR runtime for Linux. Monado aims to jump-start development of an open source XR ecosystem and provide the fundamental building blocks for device vendors to target the GNU/Linux platform.

In a recent blog post update, the team behind it talked about ways of tracking hand poses and the challenges that come along with it. So what are they doing to improve it? Hooking up Machine Learning / AI instead of just relying on vision-based methods of tracking.

[…] as part of a project backed by INVEST-AI, a program managed by IVADO Labs, we have developed a multi-stage neural network-based solution that accurately locates and tracks the hands despite complex background noise and occlusion between hands […]

Marcus Edel, Collabora

Check out their demo in the below video:

YouTube videos require cookies, you must accept their cookies to view. View cookie preferences.
Accept Cookies & Show   Direct Link

See the full post for all the details, they'll be sharing more another day.

It's impressive work, and anything that can help the accuracy is going to help the overall experience, especially while gaming. Looking forward to seeing what Collabora come up with next as Monado matures.