Three years into consumer virtual reality, the technology is still in its first generation. While minor improvements are on the near horizon, there’s a bottleneck holding back a true next generation.
That bottleneck is the development of (good) foveated rendering.
Resolution and FOV: Fundamental Enemies
Almost all consumer headsets today have a field of view of roughly 100 degrees horizontal. For VR to be more immersive, that needs to increase- human vision is around 210 degrees. But the resolution of today’s headsets isn’t good enough either- in fact, you can still visibly see the pixels and small text is unreadable.
The fundamental problem in significantly improving these specifications is that the wider the field of view, the lower the angular resolution. Angular resolution is the number of pixels per degree — this is how we actually perceive the resolution of displays. That’s why your TV looks great far away, but low detail up close.
Diagram from Oculus Connect 3
A 200 degree headset would have half the angular resolution of a 100 degree one with the same display. This means that by using a display with twice the number of pixels, you would still only get the same low detail as today’s headsets if it was spread over the full range of human vision.
Display panels with the resolution needed will exist soon — that’s just a matter of time. But the problem is in finding a way for your graphics hardware to actually drive them when they arrive.
Even if the goal is only a 50 percent increase in field of view and angular resolution, that would require approximately 4x the number of pixels drawn as current VR. The only GPU that could run such a headset at full performance on existing games would be the TITAN RTX– obviously impractical.
And if you had a headset with 200 degrees field of view and twice the angular resolution of today? That would require 16x the pixels drawn by the graphics hardware dozens of times every second. No GPU existing today could handle such a task, and at the current rate of progression it would be more than five years until one emerged. That could mean 10 years until the hardware became affordable.
Foveated Rendering To The Rescue
There’s a solution to this bottleneck. The human eye is only high resolution in the very center. Notice as you look around the room that only what you’re directly looking at is in high detail. Everything around that area isn’t as crystal clear. In fact, that “foveal area” is just 3 degrees wide.
VR headsets can take advantage of this by only rendering where you’re directly looking in high resolution. Everything else can be rendered at a significantly lower resolution. This is called foveated rendering, and is what will allow for significantly higher resolution displays. This, in turn, should enable significantly wider field of view.
Foveated rendering relies on eye tracking. In fact, that eye tracking needs to be essentially perfect. Otherwise, there would be distracting delays in detail when looking around. Not all foveated rendering solutions are created equal. The better the eye tracking, the more gains can be found in rendering efficiencies.
Pimax: A Case Study
The Pimax 8K and 5K headsets offer a case study on the need for foveated rendering. The headsets offer a resolution of 2560×1440 per eye to enable a 170 degree field of view.
Despite originally claiming the headset would run on a GTX 1070, Pimax later changed its tune to say that GPU wouldn’t be enough.
Tom’s Hardware recently benchmarked the headset using the $700 RTX 2080. While they were able to run simplistic games like Space Pirate Trainer smoothly, games like Arizona Sunshine and Serious Sam VR required turning the field of view down to 120° and setting the resolution far below native.
Pimax plans to release an eye tracking add-on for its headsets later this year, which the company claims will enable foveated rendering. We’re eager to try it out.
What’s In A Generation Anyways?
The definition of a “generation” is purely semantics. In the world of game consoles, a generation denotes a substantial improvement and happens every five years or so. But in the world of smartphones, “generations” are yearly.
To me, a new generation of VR implies a large improvement of specifications and the addition of new features. Minor resolution bumps, in my view, don’t count. When the next generation of VR truly releases, there won’t be any question about whether it fits the description.
The test I propose for generational leaps in VR is that if your long distance cousin who only visits at Christmas were to try your new headset, would they notice the difference from last Christmas? They’re unlikely to notice a minor specification bump, but they’ll notice significant improvements and the addition of new features.
So When Will It Be Available?
At CES in January HTC announced Vive Pro Eye — a refresh of the Vive Pro adding eye tracking. Vive Pro Eye will be one of the first headsets on the market with foveated rendering, though there have been developer kits in the past.
Vive Pro Eye doesn’t have a wider field of view than other headsets. Instead, it is used to enable supersampling. That’s where you render at a higher resolution than the display to get a higher quality image with less (or no) aliasing. Whether this decision was due to the quality of the eye tracking or other unrelated reasons is unclear.
At Oculus Connect 5 in October, Facebook showed off their progress on foveated rendering. The company’s chief VR researcher Michael Abrash showed off a new approach using machine learning to fill in the low resolution areas, allowing for a tiny foveal area. The example Abrash showed only required 5% of the display resolution to be rendered- a 20x saving. He claimed the result is visually “indistinguishable” from normal rendering in VR.
Abrash concluded with a prediction that this technology would be ready in four years — 2022. Based on Abrash’s other predictions in past Oculus Connects, it seems likely that Facebook is waiting on this technology to enable a true next generation Oculus Rift. In the meantime, it seems like we’ll have a mid-generation refresh to hold us over.
