Dozens of people are working at Facebook Reality Lab in Pittsburgh, Pennsylvania on research into ultra-realistic avatars that could redefine communication in the 21st century.
I spoke by phone with Yaser Sheikh, a Carnegie Mellon University associate professor running VR research in the city for Facebook since 2015. Sheikh offered context on a technical blog post released this week which details the company’s research toward ultra-realistic avatars. I followed up with Facebook over email for clarification after the call.
“Facebook Reality Lab Pittsburgh can accommodate close to 100 people,” the email explains. “To accommodate the team’s growth, FRL Pittsburgh will be moving into a larger building in Pittsburgh later this year.”
That suggests a big investment toward the future of the team. The group is also not the only one at Facebook researching more realistic and personalized avatars. To be clear, Facebook says this technology is “years away” from being realized in consumer headsets.
Facebook calls the avatar system detailed this week ‘Codec Avatars’. The work has been talked about at Facebook and Oculus developer conferences over the last few years. These latest details, however, provide insight into the work and their path which could one day carry the research into a shipping product.
Avatar Creation
To create each avatar, Facebook uses a specialized capture studio consisting of 132 cameras pointed toward the center. After a few hours with a person making various facial expressions, the system generates a unique photorealistic avatar face. Another newly-built system captures bodies.
“Creating lifelike avatars currently requires capturing large quantities of high-quality audio and video of research participants using a highly-customized system of cameras and microphones in Facebook Reality Lab’s Pittsburgh office,” the email from Facebook explains. “This data is used to train AI systems that may one day allow users to quickly and easily build their Codec Avatar from just a few snaps or videos. “
The current process won’t work with consumers for obvious reasons so a main focus of the research is to use machine learning to generate avatars of this detail. The potential is that one day your existing photos on Facebook or Instagram could be enough. I asked Sheikh if he’d tried feeding his own photos from Facebook to its system to see if he can build an avatar from it. He said they need to understand more about how light interacts with skin before trying.
”When we do such work, we want to make sure it’s with the consent of the user,” he added.
Capture Research
To drive these avatars, Facebook built a system with cameras — including one in the nose gap — on the inside of the head-mounted capture research device.
The results look impressive.
VR headsets are relatively dark on the inside and you don’t want to blind the user with light. Features can be illuminated using infrared lights that are invisible to the human eye. These types of lights are used in the Oculus Rift “Constellation” positional tracking system, and are often used in eye tracking. Facebook here is using the same technique to illuminate features in the capture system.
Other Approaches
Facebook has a separate team in Sausalito, California working on a different approach to realistic avatars.
“This involves exploring physics-based avatars that can interact with any virtual environment and combines fundamental research in areas like biomechanics, neuroscience, motion analysis, and physically driven simulations,” Facebook explains. “This technique still relies on live data capture, just like Codec Avatars, but instead of the live sensor data driving a neural network, it drives a physics-based model inspired by human anatomy.”
Facebook should share more about this approach later this year.
Long Researched, But Still Distant
Facebook’s Oculus has been researching face tracking and realistic avatars for years. At SIGGRAPH 2015, research conducted by Oculus in partnership with USC revealed a hardware prototype with an unwieldy arm attached to a development kit.
Additional sensors in the foam lining of the headset would read expressions where the camera is blocked from view.
FRL in Pittsburgh was not involved with that earlier research. Facebook is “working through possible solutions” and Sheikh shared a few “miracles” which need to happen for this to become mainstream. He said their goal, overall, is to eventually transfer the technology to the product teams. He cautioned, however, that they first need to figure out how to produce realistic avatars with a minimal amount of information provided.
”Completing the rest of the body is” another miracle, Sheikh said.
Defying Distance And Digital Doppelgangers
In a consumer product, avatars of the quality Sheikh’s team and others at Facebook are building would fundamentally change the nature of communication. No longer will we be talking to cartoon approximations with limited expression. We’ll be talking to humans, regardless of distance.
This question of reproducing nuanced human expression is outlined by some researchers and scientists as one of the hardest to solutions to find in VR. One day, though, conversations with these kinds of avatars could be the primary form of remote communication, replacing and eliminating the limitations of video calls.
The development of this technology also carries with it enormous risks. The risk of digital impersonation is huge if your appearance and mannerisms could be worn like a costume by somebody else.
“The basis of communication is trust,” Sheikh said. “You have to be absolutely certain it is me.”
Sheikh explained the point in more detail over email:
This is early stage research and as we continue to develop this technology, we’re thinking pragmatically about safeguards we can put into place to combat misuse of the avatars. For example, we’re exploring the idea of making sure your avatar is secure in an authentic account. How we work with real identities on our platforms will be a key part of this, and we have discussed several security and identity verification options for future devices. Obviously we’re a ways away from this type of technology in headsets, but we want to think about solutions now.
Staff Writer David Heaney contributed to this article.
