Well that was quick! The virtual reality community has spoken loud and clear – we want eye tracking.
FOVE – an eye tracking HMD developed by Yuki Kojima and Lochlainn Wilson, passed its $250,000 Kickstarter goal in four days. While not quite as fast as another pretty well known headset, FOVE’s rapid backing shows a strong demand for the kind of eye tracking technology that is embedded in the HMD. And it makes sense too – because eye tracking is good for a whole lot more than just shooting robots with lasers out of your eyes.
https://www.youtube.com/watch?v=uKpcbQcDItw
FOVE uses a set of infrared cameras positioned around the lenses which detect rapid movements on the eyes. Having tested them multiple times myself, I can safely say they are incredibly accurate – more so than any other eye tracking technology I have tried in the past. The HMD itself is pretty good, but isn’t quite on par with any of the majors. The display is a 2560×1440 WQHD that has a little bit of an apparent screen door effect, but it feels pretty close to a GearVR which is fine. The headset prototypes currently lack positional tracking, but the team is currently working on adding it – and it is projected for the headset’s launch. CTO Lochlainn Wilson said on reddit that the team is currently talking with Valve about lighthouse – the tracking solution used in the HTC Vive. FOVE is a solid headset on it’s own and with lighthouse tracking it could be a great one – but it’s eyetracking is really the thing that sets it apart.
Eyetracking brings quite a bit to the table for VR, so much so that it could be easily argued that it is necessary for it to be fully immersive. One of the things that I am most excited for with eye tracking is foveated rendering. Right now we get our sense of depth in VR through a process called stereoscopy – a process where an image is shown to each eye, which converge and cause a sense of depth which is actually only an illusion of depth. In reality our eyes focus on different planes of depth and that focus is one of the things that plays into our depth perception, along with perspective and occlusion. By adding eye tracking, you can detect where in an image the eyes are focusing and render the scene with correct depth focusing in real time, producing a more naturalized 3D effect. You can see the FOVE’s depth of field demo in action here:
https://www.youtube.com/watch?v=c-6dS69NaZE
In this video you can see the green dot following around the screen, this is for the purposes of helping show the focus, but that is live tracking someone’s eye position and showing you where they are focusing. In VR this technique really helps improve the sense of depth. Here is a video of the effect without the tracking dot:
https://www.youtube.com/watch?v=6qBS0EtqnzI
This technique can be used for more than just depth perception, it also brings with it a nice rendering shortcut that may help save on performance – which is great considering modern VR will require some pretty powerful hardware to run. The concept, known as foveated rendering, has been explored by researchers for a while now, including teams at Intel and Microsoft. By focusing the rendering efforts on the part of the image that you are looking at you are able to take rendering stress off the machine allowing it to render the parts of the image at the edge of your vision with less pixel density. This helps to save on precious processing power.
Foveated rendering isn’t the only trick that eye tracking enables, it also brings with it a brand new level of emotional connection with artificial intelligence. They say the eyes are the window to the soul, and it is true – eyes really play a major role in the ways we communicate with one another. In fact, the eyes are so important to communication they have their own subfield of study within the communication field. Bringing your eyes into the game also brings other character’s eyes into it. With eye tracking, AI avatars will be able to react to you with their eyes – looking at your directly when they are speaking and other subtleties of communication that make the experience feel more immersive. As AI continues to improve (and given Google’s massive investments in the field it only seems likely that it would) and with the addition of eye tracking, we could be having very real conversations with questionably real avatars soon.
Of course eye tracking also brings with it a new dimension of interactivity. For VR especially it really makes sense for things like targeting and UI interaction. In fact – as a brief aside – targeting issues in VR was one of the things that got Lochlainn so interested in bringing eye tracking to VR. Lochlainn was playing some Team Fortress 2 on the DK1 when he noticed a big issue, he was getting killed by bots. He never got killed by bots. The issue he found was because of the parallax effect in VR – which made it very hard to correctly aim at distance. Eye tracking likely won’t be used often to the extremes we saw in the lasers demo – for the very reasons outlined in this great article about design trends in VR, staring to win is too easy. But it can definitely be used in more of a targeting assist kind of way to help gamers get the aiming precision they are used to.
UI interaction with the eyes will also be pretty big. If you are a fan of the Iron Man movies, you likely have seen Tony Stark’s in suit UI system. That system is set up with voice commands and eye tracking and allows interaction with the system hands free (letting Tony focus his hands on kicking ass). The same principle would work very well in VR. FOVE has also tried some interesting experiments with typing using the eyes and an onscreen keyboard – but that may ultimately be a placeholder for better voice to text translation.
One of the experiments that FOVE has tried with eye tracking actually involves the physical world. Using their headset in combination with a piano connected to the internet they were able to enable a young boy with muscular dystrophy to play the piano on stage. You can see the video of this amazing endeavor here:
FOVE’s successful Kickstarter is only the beginning. After taking in accelerator investments from both the Microsoft Ventures accelerator and Rothenberg Ventures, this Kickstarter proves that the market is demanding this tech for VR. FOVE will likely end up taking on some additional investment on the back of this highly successful Kickstarter – just as Oculus did. Hardware development is an incredibly expensive process and it will likely require more than the Kickstarter funds if they intend to bring this technology to market as a standalone HMD, which is what they company is positioning themselves as – at least for now. I seem to remember Nimble having a successful Kickstarter as well…. But if they can get the Lighthouse integrated with their platform – and improve the screen, and I think with greater backing and investment they will, they could be a serious contender as an HMD.
Disclaimer: FOVE is currently running ads on the UploadVR website.
