Alphabet’s Google filed for a patent for a compression system specifically designed for frames produced by foveated rendering.
Foveated rendering is a process which renders most of the view of a VR headset at lower resolution except for the exact area where the user’s eye is pointed, which is detected with eye tracking. That area in front of the eye — where humans perceive the greatest detail — is rendered at a significantly higher resolution. Foveated rendering is considered crucial for future advancement of VR as it allows for higher resolutions without impossible GPU requirements.
So why compress the frame? Why not simply send the result to the headset as is?
The patent explains that in a standalone headset, the data lanes from the SoC (system-on-chip) to the display have limited bandwidth. Increasing this banwdith would have a non-trivial effect on energy consumption. Specifications like DisplayPort include an optional compression system already, however the algorithms behind it were not designed for elements of varying visual acuity in a single frame.
The new compression system described gives priority to elements within the high detail area, where the result should be “virtually lossless”. Combining the high and low detail images without visible artefacts is described as requiring a custom chip. Thankfully however this chip is described as “relatively simple”.
While the patent application was published this week, its filing date is July 2017. The patent is seemingly based on a late 2017 paper from Google Research titled ‘Strategies for Foveated Compression and Transmission’.
The project was led by Dr Behnam Bastani, who led Google’s entire VR rendering research effort. In 2018 Bastani moved to Facebook to work in the FRL division led by Michael Abrash. This seems to follow an increasing trend of Facebook poaching top VR talent from Google and Microsoft.
