Hao Li is one of the researchers behind an impressive technology demo presented by USC and Oculus Research at last year’s SIGGRAPH. The demo shows a depth sensor attached to a long nose on a Rift development kit to track facial movements. Combined with sensors inside the lining of the headset, the system shows a way of transferring a wide range of facial expressions into VR.
We know face tracking will be important for VR, but AR can use the tech right now. It’s the reason Snapchat is dominating social media, Facebook bought live filter app MSQRD and Apple bought Faceshift. We use our faces to communicate all day long in ways we sometimes don’t even realize, and advances in computer vision are gaining the power to understand even the tiniest expressions in real-time. Magic happens when you use this power to layer in digital effects. The most immediate market for this type of thing is entertainment, but the technology has enormous potential beyond that.
As I was looking into face-tracking solutions recently I wanted to find out what happened to Hao Li after presenting at SIGGRAPH last year. Turns out he’s developed a technique to track facial movements in real time using a “single RGB input” — no depth sensor needed. According to Li, the company he founded, Pinscreen, is funded to the tune of $1.8 million in seed money from Lux and Colopl VR Fund among a number of tech industry executives.
While it’s still very early and the company is in “stealth” mode, raising so much in seed funding is a strong endorsement of the approach. Demo videos show the technology recognizing head movement and facial expressions despite hair, sunglasses or hands blocking parts of the face. While it’s being shown in action doing faceswaps and modifications similar to MSQRD or Snapchat Lenses, Li said in an email they aren’t planning to build just another version of those apps.
“It will be something a lot more addictive,” he wrote. “We have a new technology that can build a complete avatar including hair from a single image, using a new deep learning based algorithm.”
Pinscreen’s Facebook page shows offices in Santa Monica, which is relatively close to where Li works as an assistant professor at USC. This looks like another startup to keep an eye on as a lot of very large companies are hoping to integrate this kind of technology.
Here are a few more demo videos:
