Quest Store & App Lab Apps Can Now Use Hand Tracking and Quest 3 or Quest Pro Controllers Simultaneously
Learn about the new Multimodal feature that allows developers to integrate hand tracking and controllers in Quest Store and App Lab apps.
Introducing Multimodal: Hand Tracking and Controller Integration
Quest Store & App Lab apps now have the capability to utilize hand tracking and Quest 3 or Quest Pro controllers simultaneously. This new feature, called Multimodal, was initially available as an 'experimental' feature in the Meta XR SDK. However, with the release of version 62, it is now fully accessible to the public.
Previously, developers could incorporate both hand tracking and controllers in their Quest apps, but there was a delay when switching between modes. Additionally, when using hand tracking, the controllers would disappear and cease to be tracked. With Multimodal, there is an instantaneous transition between hand tracking and controller tracking, allowing for a seamless user experience.
Mixing Hands and Controllers
One of the key advantages of Multimodal is the ability to mix hands and controllers within a virtual environment. For instance, games that only require one controller can now track the user's other hand as well, if the developers choose to implement this feature. This opens up exciting possibilities for games like GOLF+ and Walkabout Minigolf, enhancing the gameplay and immersion.
Furthermore, developers have the option to attach controllers to objects or body parts, effectively using them as generic trackers. However, this functionality is more optimized for Touch Pro controllers compared to Touch Plus controllers, as the latter may be frequently occluded from the headset's view.
Compatibility and Limitations
It's important to note that Multimodal is only compatible with Quest 3 and the Touch Pro controllers. Quest 2 controllers are not supported. Quest 3 is capable of running hand tracking alongside controller tracking simultaneously, filling in gaps when a controller is temporarily obscured. On the other hand, Quest 2's chipset may not be able to handle both functionalities concurrently without sacrificing performance.
Enabling Multimodal also comes with certain restrictions for developers. It prohibits the use of other features such as Inside-Out Body Tracking, hand tracking Fast Motion Mode and new Wide Motion Mode, as well as tracked keyboards. Developers are advised to consider these limitations when implementing Multimodal in their apps.
