Hands-On: Facebook’s New 24 Lens Camera Turns Real Life Into High Quality VR

by Joe Durbin • April 19th, 2017

Most 360 content isn’t very good. Whether or not you consider this content to be “true” VR, the truth is much of it isn’t even worth debating. For every standout from a team like Felix and Paul, there are 20 low-resolution examples overflowing with meh. But even the best ones now look like garbage compared to what I just finished seeing at F8.

Hidden in a side hall of the San Jose Convention Center, and shown by invitation only, were Facebook’s and FLIR‘s brand new cameras. The x24 and x6 were both on display in the Facebook Surround 360 demo chamber. After being ushered past a series of security guards I emerged in a room divided by a few temporary walls. In the center were the two cameras in all their spherical glory.

facbook-surround-x24-x6

Representatives from the Facebook Surround 360 team brought me into the first phase of what would become a marathon of demos. Step one was to see what the x24 can produce inside an Oculus Rift.

Inside the Rift I saw what appeared to be a standard, but very high resolution 360-degree video. I was standing inside on the balcony of a massive room housing what appeared to be a lush indoor rainforest. From what I could hear a river was flowing somewhere and leafy trees swayed playfully in the breeze. I was impressed by the high image quality and then my handler informed me he would be turning on the true magic of the Facebook Surround experience: true freedom to move around.

With the push of a button my world exploded outward. Suddenly the palm tree I had been admiring was inches from my face rather than part of a two dimensional wall of color. I could suddenly tell just how high up I was as the ground fell away. All around me reality had gone from a flat plane to a vibrant swath of depth and color. In all of this, the image clarity remained incredibly high resolution. The only glitch was a stitching bug that placed a sort of blur effect on certain edges.

facbook-surround-descriptions

Here’s where things get really cool. Just as I was getting used to my surroundings I was instructed to take a step. As I did so the world stayed exactly where it should while I started to move. For the first time in my life I was walking through a 360-degree video and I couldn’t help but smile. Honestly, if it wasn’t for the occasional blur or glitch, I would have believed I was truly standing in this indoor arboretum.

From here I was shown an additional scene featuring a group of tourists ogling a tropical aquarium as an instructor explains the fish. Again, I was shown a comparison between a traditional, flat 360-degree video, and Facebook’ new walkable, 3D experiment. The difference is remarkable. It’s all of the presence-inducing factors that make a CG experience built in Unity believable, but with high-res video instead of assets from the Unity store.

This isn’t to say that Unity is getting left behind. The next stop on my demo tour was with OTOY, one of the Facebook Surround team’s early partners. OTOY showed me a simple park scene captured by the x24. Once again, the fidelity was incredible and the three dimensional improvements are outstanding. However, the real kicker is that OTOY was rendering all of these photorealistic images with Unity in real-time. This meant that I was able to alter what I was seeing with basic Unity lighting tools.

With a push of a button on the Oculus Touch controller I turned off the sun and started experimenting with new light sources and angles as I manipulated a white globe around my environment.

In this moment I was able to apply video game level editing features to a video of the real world. 

Here’s OTOY CEO Jules Urbach showing the concept:

If this were applied to actual game design, then we might one day see vast open worlds created using actual captures of reality rather than computer generated equivalents.

Anshel Sag, an analyst with Moor Insights & Strategy also tried these demos today. He also notably got his hands on HypeVR’s tech as well, which has a very similar volumetric video demonstration. Sag says of Facebook’s current offering:

The Facebook volumetric video solution is clearly in its infancy but with their partnership with OTOY they expand the usefulness of the volumetric video and images in ways that we have yet to comprehend. In terms of visual quality and 6-DoF the Facebook 6-DoF solution still has a ways to go to be where other 6-DoF video solutions like HypeVR are today. Admittedly, the HypeVR rig is quite large and expensive, but today it produces a superior 360 6-DoF video experience. Because Facebook has only been at this for 10 months, I suspect there’s still lots of improvement coming and they can eventually reach what HypeVR has today.

It will be interesting to see how Facebook brings these cameras to market because the 24 camera rig is built by FLIR and the 6 camera rig is fully internally made by Facebook. The choice not to open source these cameras also means that Facebook intends to license out their design to prospective camera companies that want to offer Facebook’s technology. I believe we will start to see content from there cameras around Q4 this year or Q1 next year as they took chains for the cameras become available.”

The realities of marketing, distributing and creating this kind of content still remain to be seen. But as it stands now Facebook Surround 360 looks like it could be the most effective camera platform we have seen yet, and a huge step toward improving the overall fidelity of this rapidly emerging industry.

Tagged with: ,

  • Duane Aakre

    Any word on the pixel density of these cameras? Current 4k 360 videos give approximately 10 pixels per degree, which aligns with the 10 pixel per degree resolution of the Rift and Vive. It seems possible we will be seeing 20 pixel per degree headsets next year. Will these new cameras be compatible or will they be quickly out of date?

    • Jules Urbach

      The 6DOF point cloud from x24 has about 120 million 3D RGB points it outputs per frame, and is not stricly a video frame you can map to a texture. If you asked what the pixels/degree of the stanford bunny model was I would reply that it’s more about the resolution of the mesh than the pixel density it’s rendered at.

      But if we take the 6DOF data as a source and flatten into a fixed mono/stereo 360 pano (er/cube) video like you were referencing, we seem to reliably get 1:1 mapping of novel rays at ~20+ pixels per degree (in my testing so far so just a WIP estimate). This resolution is also true for novel viewpoint rendering in 3D space near the camera origin when playing back a fulll 6DOF deep pixel video stream on an HMD w/ pos tracking.

  • Ben

    Why keep the nose turning “360” moniker in the product name when it has 6DoF capabilities? I almost didn’t read the article after seeing “surround 360” printed on the cameras.

    • hellol11

      Agreed. 360 content was made easy to produce thanks to google, it’s the ability to move around in these environments that should sell these things.

  • Cool! Jaunt videos too have depth… can you make a comparison between a Jaunt video and a Facebook video?

    • Jules Urbach

      I do not have enough info to make a fair public comparison between the two. I will say that the FB depth estimator stack is the best I’ve used so far inside our workflow. I think FB video team will improve the DE a lot even before the camera hits the market. We also have some ideas on how to maximize quality when it gets packaged into an ORBX scene for re-rendering.

  • polysix

    Sorry but the phrase “facebook” combined with “24 lenses” has ZERO place in my future. You’d have to be fucking lobotomized to allow those pricks anywhere near you with cams!

    • hellol11

      I can guarantee you that that’s already happened god knows how many times. Whenever you take a selfie and upload it to Facebook, it knows it’s you. Same goes for a picture someone else took. They don’t even have to get near you with cameras, they just have to wait for literally 10 seconds, and look, it’s you, on Facebook again.

      Also, you upvoted yourself. You’re “special”, aren’t you?