Facebook’s Oculus patented an eye tracking technique which uses light field cameras inside the headset. Most previous eye tracking systems used a regular or infrared camera combined with an IR illuminator to keep the eyes lit.
A light field camera differs from a regular camera in that it also captures the direction that light is travelling. This directional information can be used to understand the depth of the image, and thus 3D shape of the eye, instead of just the color and brightness. By knowing the 3D shape of the eye, the system can find out where the pupil is relative to the eye itself, and thus a more accurate estimation of the user’s gaze direction than with just the apparent 2D shape of the pupil.
Eye tracking can greatly enhance the feeling of “social presence” in multiplayer VR, but its most promising use case is foveated rendering. Foveated rendering is when only what you’re looking at is drawn at full resolution while the rest of the scene in your peripheral view is rendered in low detail. This works because human vision is only high detail in the very center. To see this for yourself, look at some text in the room you’re in right now then look just a few feet to the side of it and try to read it again.
Foveated rendering should one day enable much higher resolution VR headsets without requiring an expensive top of the line graphics card. Finding a way to make it work reliably is crucial to the future of VR.
At Oculus Connect 3 in late 2016 (video above), Oculus Chief Scientist Michael Abrash discussed the challenges of developing an eye tracking system good enough for foveated rendering. He stated that for foveated rendering to be invisible to the user, “virtually perfect” eye tracking would be needed. If eye tracking fails even for a fraction of a second, the user would see a dramatic reduction in visual quality. Even today, no company has yet shipped this quality of eye tracking in a consumer product.
In late September this year at Oculus Connect 5, Abrash stated that he expects this quality of eye tracking to be ready within four years (by 2022). Perhaps this estimate is dependent on having a real-time 3D view of the eye using a light field camera, as described in this patent. If true, it would make eye tracking more expensive to integrate than many had hoped, but would likely result in a dramatic improvement in reliability.
