Maintaining full framerate in VR is so crucial that VR companies have developed special driver-level techniques to compensate when frames are dropped. On Rift, Oculus uses a technology called Asynchronous SpaceWarp (ASW), which, when framerate isn’t being met, forces the app to drop to half framerate and fills in every 2nd frame with a synthetic one extrapolated from the previous. Valve and Microsoft also have similar technologies for their respective Steam and Windows platforms.
Where this technology would be even more useful, however, is on mobile, where due to the much lower compute power it is much harder to maintain framerate. This would be especially useful for the upcoming Oculus Quest all-in-one VR system, where the focus is games, and which Oculus is trying to encourage developers to port their PC VR games to. The Oculus Mobile (Gear VR, Go, and Quest) runtime already performs rotational reprojection (called “Asynchronous TimeWarp” / ATW), but this ignores positional head movement.
At Oculus Connect 5, engineer Jian Zhang discussed the limits of rotational-only reprojection in a 6DoF environment like Quest, noting that it leads to positional tracking judder from the incorrect projection. He then spoke of an experimental fix in the works called ‘Motion Reprojection’, which would use the depth buffer submitted by the app each frame in order to more accurately reproject the view with respect to the user’s head position.
Essentially, if a frame is dropped, the Quest’s software would generate a synthetic alternative frame by skewing the image in the rotational direction (which Go and Gear VR already do), and now positional direction, that the headset moved. Having the depth information from the scene allows this new positional reprojection.
This approach would, however, not be fully equivalent to ASW on PC. Whereas ASW also takes the animation and motion of objects in the scene into account and extrapolates them (by reading the color buffer), ‘Depth Reprojection’ would only be making head tracking feel smooth – objects in the scene would still have animation judder if the app was rendering at a low framerate. CTO John Carmack stated on Twitter that doing color extrapolation like ASW is “actually more work than most mobile frames to render”, so this is unlikely to come to Quest any time soon. Zhang also noted that Depth Reprojection would not deal well with transparency such as a virtual fish tank.
Oculus have not given any details on when ‘Depth Reprojection’ will be coming to Quest, but if it does, it would help the headset to handle complex scenes in games. Developers would still need to put serious effort into optimization to avoid animation judder, but at least a temporary drop in framerate will not make the user physically sick through head tracking judder like was seen in the Oculus DK2 days of PC VR.
