Tracking the intricate movements of your face is a major area of investment by a number of companies, from Snapchat and MSQRD’s filters to Apple’s Faceshift and the major performance gains possible with foveated rendering.
With virtual reality headsets, transferring the full emotion of the human face is going to be challenging. A creative app like Mindshow, for example, lets you embody characters whose facial expressions are changed with the touch of a button on a controller. If you want to develop VR experiences that depend on facial expressions, using a controller to tell the computer your expression is the kind of work-around needed until face tracking gets cheaper and easier.
One of the coolest approaches we’ve seen so far was demonstrated last year through a collaboration between USC and Oculus Research (PDF). It featured a depth camera on a long nose accessesory that gives it a good angle looking at the mouth. Strain gauges inside the lining of the headset measure upper facial movements. Together the two methods provide an accurate model of a variety of facial expressions despite much of the face being blocked off.
BinaryVR has had a face-tracking demo readied since December of last year that is said to recognize 20 different facial expressions using a camera hanging down in front of the headset. Check it out here:
I tried the demo briefly at VRLA and it recognized a smile, me pursing my lips into a kissing motion and moving my jaw and chin to either side. At the same time, the camera missed a frown I made. BinaryVR, which is based in Burlingame, is launching $349 pre-orders today for the 3D camera on its website, with plans to ship it in October with a mount you can attach to a consumer headset.
If you’re a developer trying to build VR software that relies on facial expressions as an important part of the experience, there aren’t a lot of options for getting started. Veeso launched a Kickstarter for a face-tracking headset and SMI now offers an eye-tracking developer kit for the HTC Vive. We hope there are a lot more accessories on the way for developers to explore next generation enhancements to VR.
If you know of other face-tracking development kits available to developers, or if you are developing face-tracking software, let me know by emailing ian at uploadvr dot com.
