Mark Zuckerberg was interviewed by Lex Fridman in VR, with both using Meta's prototype photorealistic Codec Avatars.
Codec Avatars is Meta's long-running research project aiming to revolutionize remote communication by achieving truly convincing photorealistic real-time avatars driven by headsets with face tracking sensors, and eventually glasses too.
Meta first revealed the project's existence in 2019, and has showed off multiple fidelity updates in the years since and a full-body version. Powered by machine learning, the avatars are generated using a specialized capture rig with over 100 cameras, but Meta researchers are working to replace this with a smartphone scan.
Zuckerberg and Fridman were both wearing Quest Pro, which features built-in face tracking and eye tracking. However, it's unclear whether the avatars were being decoded and rendered onboard, or whether a PC was doing this. A USB-C cable is connected to both headsets, but this could just be to avoid running out of battery during the interview, rather than a connection to PC VR.
Throughout the interview Fridman frequently expressed feelings of shock and disbelief at what he was experiencing, claiming it truly felt as if he was sitting across from Zuckerberg, at one point saying he genuinely forgot that the avatar he was looking at wasn't a real person. He declared that unlike almost all previous attempts at photorealistic digital humans, Codec Avatars surpass the uncanny valley - as we noted in our hands-on last year.
Of course, Meta still has a long way to go to reach this level of fidelity in shipping products. The widely ridiculed Meta Avatars today have a basic cartoony art style that couldn't be further from photorealism, and Quest 3 has neither eye tracking nor face tracking sensors. But if a leaked Meta roadmap from earlier this year is to be believed, Meta could be aiming to deliver Codec Avatars in the next version of Quest Pro, which it reportedly plans to ship in 2025 through a partnership with LG.
