Foveated rendering, a technique that mimics human vision, could help catapult virtual reality to the next level, but it is something that to this point hasn’t been achievable with current technology – until now.
SensoMotoric Instruments (SMI), one of the oldest and largest companies focused on technology that tracks the human eye, is planning to show off foveated rendering at CES next week alongside the newest edition of their technology – a 250Hz eye-tracking kit for HMDs.
https://youtu.be/Qq09BTmjzRs
For those of you who aren’t jumping for joy already, foveated rendering is a big deal for virtual reality because it could enable us to create realistically rendered experiences using today’s GPU and CPU technology. Foveated rendering takes advantage of human vision to cut down on what actually needs to be rendered to the screen. When you look at an object only the center of your focus, where your eye’s fovea is, appears perfectly clear with the image getting progressively less so as it approaches your peripheral. Foveated rendering takes advantage of this by using low-latency eye tracking to track the eye’s fovea and then only renders full resolution of the image where the user’s focus is. The benefit being that the computer has significantly less to render because it only needs to output full resolution where you are looking, thus enabling much greater performance with current hardware.
Just how much greater? According to SMI OEM Director, Christian Villwock, “a factor of two to four is easily achievable right now, and with more effort you can even achieve much higher factors of improvement.”
With that sort of performance improvement, getting modern computer systems to render VR to upcoming 4K, 8K or even 11K screens becomes much more of a possibility. Which is important because in order for rendered resolution to match real world resolution, you need to render at least 8k in each eye according to Oculus’ Chief Scientist, Michael Abrash.
“We are working on a successor project with a few of these companies on the PC side and also on the mobile side”
SMI achieved this through a mixture of “a lot of optimizations” on the software side, alongside a “close collaboration” with the image sensor company, OmniVision. Previous attempts at foveated rendering broke immersion, as they were not fast enough to keep up with your eyes’ rapid movements, resulting in you seeing a little circle of crisp resolution trailing your eyes’ path. SMI’s “low-latency algorithms” work in conjunction with OmniVision’s cameras which operate at a 250Hz refresh rate to enable foveated rendering that Villwock says “is not preceivable anymore.”
“There is this magic number of 240Hz that we have found through over ten years of research,” says Villwock, “at this rate you can start measuring saccadic eye movements, that is very important.”
He is referring to the rapid, simultaneous movements of your eyes as they fixate on a single point. Even if you are staring at a single point, your visual fixation is never actually steady; rather your eyes are constantly involuntarily moving at a fairly rapid rate. You can see this demonstrated in the video above which shows someone watching a monitor in front of him or her. Measuring these movements allows for vastly more accurate measurements of where the eyes are actually focused, enabling things such as foveated rendering.
So far, SMI has only shown off eye tracking integration as part of an upgrade kit for the Oculus Rift DK1 and DK2, but that may change going forward.
“We are working on a successor project with a few of these companies on the PC side and also on the mobile side,” Villwock told UploadVR. He declined to elaborate further on those projects but did tell us “wait maybe two months or so and we will be able to talk more.”
Eye tracking technology is something that is going to be a “critical part” of the future of VR technology, according to Oculus founder Palmer Luckey. Eye tracking enables much more than foveated rendering, it can be used to help create a true sense of depth, to create better user interfaces, or even to enable much more engaging social interaction in virtual spaces. That is why so many companies, including Oculus, are actively researching the technology.
“My personal belief is that all version two headsets will have eye tracking integrated,” Villwock told UploadVR.
SMI will be demonstrating this potentially revolutionary breakthrough in eye tracking technology at CES. We plan to go hands on with it to verify these claims for ourselves. We will be sure to report back with more details.
