Facebook continues to evolve its spectator tools for virtual reality, but creating mixed reality footage remains an arduous process from the standalone Quest while the company continues to actively explore the approach.
The company recently added a “Live Overlay” feature in the most recent v29 software update for Quest which merges the first-person view in VR with a camera view of the person wearing the headset. The feature is starting to roll out to a limited number of users in the Oculus app “and it will work with any VR app that supports Casting and Recording,” Facebook explained over email.
In addition, a page in the Oculus Quest developer documentation explains “Spectator Camera is extending Mixed Reality Capture (MRC) to the Oculus app casting.” We contacted Facebook to ask for clarification about this feature because the language in this description isn’t clear. The “Spectator Camera” will allow casting the view from VR to the Oculus app while enabling control of the view from the phone using touch controls, Facebook explained. It was announced at Facebook’s last VR developer event but doesn’t include a view of the VR user wearing the headset.
“There aren’t any apps live with this today, but we are actively working with developers to integrate, so that’s why these developer tools are available now,” Facebook explained.
And while the “Live Overlay” feature does feature an external view of the person in VR, it doesn’t include a third-person angle or the careful alignment that’s key to “Mixed Reality Capture.”
Mixed Reality Capture is driven largely by community projects making it possible to merge simulated content streaming from a VR app with an actual camera’s view of the scene. The merging of two realities becomes a single seamless view that’s instantly understandable to spectators. In 2016, Valve essentially popularized the idea with its initial promotional video for SteamVR. Since then, community projects like LIV and Fabio Dela Antonio’s Reality Mixer app enable developers and broadcasters to more easily share what it feels like to play VR games without post-production work. Some of these projects, however, require an arduous alignment process to sync the view of two realities, or requires the addition of extra recording software like OBS.
Dela Antonio maintains an open source repository for his work on the Reality Mixer app and Facebook CEO Mark Zuckerberg this week posted a mixed reality video of himself playing Beat Saber using what appears to be the tool. We reached out to Dela Antonio for comment on the Zuckerberg video and to see what Facebook’s interest in mixed reality broadcasting means for his work. He wrote to us over direct message that he thinks Zuckerberg’s video and the developer page are evidence the company will eventually offer a solution similar to his app.
These buttons in the top left corner of the video look familiar 👀 https://t.co/7F4xmJRdML pic.twitter.com/P5t2eSJffA
— Reality Mixer (@reality_mixer) May 25, 2021
“I generally see this as a positive thing, it means that these videos will become even more accessible, and that they’ll start making some improvements to the existing Oculus MRC mechanism,” he wrote. “As for my app, its future compatibility with the Quest will depend on the changes they make to the MRC mechanism. I’m still working on the app as normal, I’ve recently added support for Avatars using ARKit’s body tracking and I’m also working on supporting physical green screens…there are some other things I could do to improve the calibration process and to allow the user to move the camera around the scene…but that’d require changes to how the Oculus MRC protocol works, it’d be interesting to know if Oculus has similar plans/ideas…One reason I was able to start working on my app last year was because the Oculus MRC plugin for OBS was open source, and that’s also why I’ve kept my app open source. It’d be nice if they could open source their new mobile Mixed Reality solution…”
Another @tiltbrush experiment. This time @gmarques33 and I used the #OculusQuest2 with @reality_mixer to record ourselves in #AugmentedReality while playing with #OpenBrush. I’d say our drawings are ok, considering that we’re both software engineers 😅 #VR @IcosaGallery pic.twitter.com/MJRgdDY711
— Fabio Dela Antonio (@fabio914) February 21, 2021
Facebook offers the Portal line of video calling devices which include a wide angle camera and a key feature of the hardware zooms in on people in the scene. But years after release, there’s still no integration with Oculus Quest for Casting or any other functionality. Recent tweets from a Japanese group shows how full body tracking might be possible with an external camera and Oculus founder Palmer Luckey recently commented that Portal should be used for the same purpose.
“Overall, we are exploring a variety of ways to help people better capture, represent and share their VR experiences. We believe these kinds of tools help to make VR more fun, and are also good for the ecosystem since they enable people to share the experiences they’re having in VR,” Facebook shared in an email. “Moving forward, you will see us continue to experiment with different features and capabilities in the Oculus App that provide new ways of casting, recording and sharing the VR experience with various different views of reality, virtual reality, camera perspectives, etc. There are many ways that it’s possible to represent and share VR, and every VR app is different – That’s why we’re testing different approaches to learn from our community about what modes of capture best represent them and the VR apps they use. In the future, features like these could possibly evolve into a mobile ‘mixed reality’ casting/recording experience like you describe. We don’t have specific plans or timing to announce for that, but it’s something we are exploring – in fact, we showed a proof of concept for this at OC6, so it’s something we’ve been continuing to explore over time.”
