Godot Engine Details Recently Improved VR & MR Support

The Godot engine has recently made significant improvements to its support for VR and mixed reality.

If you're not familiar with it, Godot is a free and open-source alternative to Unity and Unreal. It's technically controlled by the non-profit Godot Foundation, but all development takes place in the open.

A breakdown of the improvements was shared by Bastiaan Olij, the core developer for XR support in Godot. Olij says "several" of these improvements "benefited greatly" from the financial and technical support from Meta announced earlier this year, with the results "improving the experience not just for Meta Quest users, but also for other XR systems".

It comes two weeks after a Meta engineer released a port of the Godot editor itself for Horizon OS, enabling the development of full games completely standalone on Quest 3 and Quest Pro headsets.

Here are all the recent Godot XR improvements Olij outlined:

XR Architecture Improvements

Improved Frame Timing

"When OpenXR was originally implemented in Godot 3 there were limitations on the order in which certain actions took place. Most of the communication with OpenXR was tied into the rendering engine resulting in Godot relying more on predictive algorithms to position tracked entities such as controllers. Originally Godot 4 inherited this approach.

As part of the Godot 4.3 release much of this logic was redone to interface with OpenXR correctly. Godot now correctly informs OpenXR when it starts processing a new frame and obtains pose data for tracked entities with predicted positioning for the current frame.

Standardization Of Seethrough & Passthrough AR

"Passthrough is a mechanism where video cameras on a VR headset record the real world and present this real world view as the background inside of our HMD turning the headset into an AR device. Different headsets implement this feature in different ways and this caused difficulties for developers who were targeting multiple devices.

While passthrough allows some neat things not possible with optical AR devices, for most AR focussed application developers should not be tasked with writing different code to support all these devices.

We have thus standardized this approach through utilising OpenXRs environment blend modes and moving platforms specific code into the OpenXR vendors plugin."

Standardization of Hand, Face & Body Tracking

"Hand tracking has been supported in Godot for a long while now, however improvements have been made to ensure better cross platform support, including standardisation between OpenXR and WebXR. The OpenXRHand node has been deprecated, in favour of the new XRHandModifier3D node which is designed to work with any XR system - not just OpenXR.

Face tracking support has been added with face tracking data being sourced from several devices. Godot now supports the Unified Expressions standard for handling face tracking data.

Body tracking support has been added using Godots Humanoid Skeleton as a base which allows various body tracking devices to leverage Godots retargeting logic.

While these 3 systems have been implemented under the XR name, their use cases are not limited to XR. Already several plugins have been created using this system to bring in body tracking data from various body tracking solutions ranging from optical Mocap to full body tracking suits."

Improved Vulkan Foveated Rendering

"The compatibility renderer uses OpenXRs foveated rendering support directly but this comes with various limitations and is not applicable outside of OpenXR.

In the Vulkan renderer we have had Variable Rate Shading support for a while which allows foveated rendering to be applied in Godot without the same drawbacks.

In the latest release various improvements have been applied that provide more control over the density map used and allow for eye tracking to impact the density map and ensure quality rendering at the users focal point."

Improved WebXR Support

"WebXR support in Godot is seeing continuous improvement, most notably the addition of hand tracking support, but also support for MSAA and a number of smaller bug fixes and improvements. Our WebXR implementation now also supports AR applications."

Meta-Specific Improvements

Enhanced Passthrough & Hand-Tracking Support

"On top of the standardisation of passthrough and hand-tracking in Godot, support for many extra features Meta provides was implemented in the Godot OpenXR Vendors plugin.

For passthrough, this includes applying various filters to the passthrough imagery, like adjusting the brightness, contrast and saturation, mapping the colors to other values, or even projecting passthrough onto custom geometry, in order to bring only specific objects from the user’s physical environment into an otherwise VR experience. See the Meta Passthrough tutorial for more information.

For hand-tracking, developers can now access hand meshes, collision shapes for the hands, and do some simple gesture tracking, like detecting pinches between the user’s thumb and any of the other fingers. See the Meta Hand Tracking tutorial for the full details about these features."

Composition Layer Support

"Composition layers are a standard feature in OpenXR, that allow developers to add floating “panels” (either flat or curved) that have 2D content rendered onto them from a Godot SubViewport.

The result of this approach is far sharper detail of the 2D content compared to applying this same content as a texture on a 3D mesh.

This is ideal for in game menus, media viewing, or heads up displays. Small text is especially more readable when using this technique.

W4 Games added both the core implementation and Meta-specific extensions, which allow applying further sharpening or supersampling, advanced alpha blending or marking particular composition layers as containing “secure content”. See the Meta Composition Layers tutorial for more information about these Meta-specific features."

Scene Discovery & Spatial Anchors

"These two closely related features are possibly the most exciting part of the work done so far. Meta’s Scene Discovery allows the Quest to scan your environment and provide your game with information about the real world.

This can allow you to place objects on your walls, on furniture, or have NPCs interact with the real environment.

Spatial anchors allow you to make this persistent. Create a spatial anchor that references a place on your wall, and you can show a virtual screen there anytime your user comes back to your application.

See the Meta Scene Manager and Meta Spatial Anchors tutorials for more information."

New/Improved XR Device Support

Pico 4

"Godot has had support for PICO headsets for while but in the past year improvements were made to this. PICO has fully adopted the official Khronos Android Loader for OpenXR and deprecated their custom loader. Godot fully supports the new approach."

Magic Leap 2

"Support for the Magic Leap ecosystem was recently added to the vendor plugin as well. Magic Leap stands apart from other standalone Android devices being an optical AR device and running AMD hardware. On these devices you need to use the Vulkan mobile renderer."

What's Coming Next

Olij says we should "stay tuned" for more new improvements and features in the near future, and shared some of what's being worked on.

Separate Rendering Thread

"There is more work that needs to be done in Godot 4.4 to make this fully viable but the foundations are in place.As part of implementing the improved frame timing logic, we did a large cleanup of the XR code to make sure it runs properly when rendering happens on a separate thread. Especially for XR applications this brings us closer to ensuring a steady framerate, even on stand alone devices."

Logitech MX Ink Support

"Not a new headset, but a pheripheral, the MX ink is a positionally tracked stylus that can be used with a Meta Quest. Support was merged into master recently and this device will be supported in the upcoming Godot 4.4 release. We’re looking into supporting this in a future 4.3 patch release."

Standalone HTC Vive Headsets

"HTC provided the team with HTC Vive XR Elite units allowing us to ensure Godot runs on HTC standalone devices. We’re also working together with HTC to improve the experience on these devices."