One of the co-founders of groundbreaking Los Angeles-based VR startup Kite & Lightning received his new iPhone X on Saturday. By Sunday night, Cory Strassburger produced a proof-of concept using the phone’s landmark real-time facial recognition system to animate a model for a character from their upcoming game, Bebylon: Battle Royale. While the proof-of concept is not perfect, there appears to be enough room for improvement achievable with the approach that he can use it for his specific purpose. A real-time facial capture system embedded in your phone could dramatically lower the cost to make digital characters look more alive and expressive in a wide range of content, including VR and AR software.
For those unfamiliar, the premise of Kite & Lightning’s Bebylon is a future in which immortality has been achieved. The drawback is that newborns stop aging at the baby stage. Thus, a society of “bebies” that live forever spend their days entertaining themselves with a vehicular combat game focused on over-the-top attitude and spectacle. These bebies have enormous personalities, and the iPhone X face capture system — the same one used to make animated emoji — could be how they are brought to life.
“I need fast and easy,” Strassburger wrote in an email. “The free bonus would be I can easily make a super cool AR promotional app where Bebylon players (or anyone for that matter) can capture some fun clips with their Beby character… and eventually, when this tech is more widely adopted, have players record expressions into their ‘character’s expression library’ using a complementary mobile app, that can then be uploaded to the game and used by them in gameplay for funny taunts and such.”
It’s still an early test and there may be further improvements he can achieve, but Strassburger wrote that “honestly its better data than most anything I’ve seen when it comes to real-time” capture and “I’m pretty confident that I’ll use this process for our game, cinematics and marketing content.”
Check out the video below where he does a pretty good impression of Christopher Walken while animating one of these bebies.
Apple’s iPhone facial recognition system is based on its 2015 acquisition of startup Faceshift. The startup showed how it could animate digital characters in real-time without any setup. The tech also made use of PrimeSense, another Apple acquisition that originally built the motion capture system in Microsoft’s Kinect. It appears that over the last several years Apple has been totally focused on miniaturizing the tech so that it could be used in a device as small as an iPhone. Over the next few years it is likely we’ll see the technology proliferate across Apple’s entire product line.
Using the iPhone X facial capture in combination of different technologies including Eisko and an Xsens motion capture suit, Strassburger believes there is an “an incredibly cheap” capture pipeline for human performance “that is super fast and easy to set up with very impressive results.”
Follow Kite & Lightning on Twitter for future updates, and we’ll keep following as well.
