Earlier this year Lytro, the company behind the world’s first consumer light field camera, made a monumental $50 million pivot into the VR space.
At the time, the company was rather quiet as to what would come of that pivot but that didn’t stop our own Ryan Damm from sifting through the tea leaves to piece their plans together. In his article, Damm predicted that Lytro had pivoted with that money to create a light field camera for VR: “imagine Lytro produces a ‘camera’ that’s essentially a holographic box; it captures all the light rays entering a defined space. Using creative 2D slices, you could synthesize any perspective from within that box (and perhaps some beyond the box, looking back through it, with limited field of view).”
Well today, Damm gets to look like a prophet because that is exactly what Lytro has been working on, the world’s first live action light field 360º camera. Dubbed “Lytro Immerge,” Lytro’s new camera and the technology stack to go along with it represent a potentially massive step forward for high-end VR content creation.
Read More: Just what the heck is a light field anyway?
Right now in the virtual reality world, there is a lot of debate as to whether 360º video is actually VR. Ask Alan Yates, Valve’s Chief Pharologist, what he thinks about it and he will give you his honest opinion, and it isn’t a favorable one. But if you ask him about light field capture, well, that’s a whole different story:
@JasonZnack nope, lightfield is far more compelling.
— Alan Yates (@vk2zay) September 15, 2015
Light field capture brings a dimensionality beyond anything we have seen with spherical video. Many companies, like Jaunt and NextVR for example, have claimed to capture light fields. Their cameras are able to add a degree of parallax, the ability to shift perspective, which while awesome isn’t really a light field. (For more on why that is be sure to check out Damm’s article on the subject).
So, what makes Lytro Immerge different?
Unlike standard 360º video, light field video captured with Lytro Immerge allows for six degrees of motion freedom within the camera’s volume, which is about one meter. This means that you could conceivably move around within the volume of the sphere and lean in toward objects. In addition to adding a degree of positionally tracked volume to the scene, a true light field recording like Lytro’s creates both horizontal and vertical parallax, giving the scene true depth and perspective regardless of your viewing angle.
We have seen some awesome real world light field captures before from OTOY, however those images were still. In contrast, Lytro Immerge is able to capture a full 360º light field in motion at “high cinematic frame rates.”
So how exactly does one do that?
Lytro Immerge uses a modular camera system consisting of “hundreds” of individual cameras in rings, each armed with Lytro’s unique image sensor. These image sensors are where a lot of the magic happens: a light field array of light field cameras, each recording the direction and position of light across a full 360º. The fidelity of the final image depends on just how many rings of cameras you decide to use and the density of the sensors inside each of them. Higher fidelity can mean higher positional fidelity, so your view is ‘better’ when you’re moving around. It can also mean higher resolution in the headset, “up to 2k per eye” according to Lytro.
When designing the system, Lytro wanted to focus on making it highly configurable. “You have to think of architecture that allows for flexibility,” says Ariel Braunstein, Lytro’s Chief Product Officer. By ‘configurable,’ Braunstein means you can deliver to lots of different formats from the same light field data: you can process it into a 180º or 360º view, monoscopic or stereoscopic, or use the entire light field and get positional tracking too – all from the same footage. Even more, that monoscopic data can be streamed to a live monitor, something vitally important for professional content creators.
In addition to controlling how the data is rendered, you can also control how the data is captured by adjusting the camera itself. The camera consists of a number of modular rings you will be able to add or remove, allowing you to scale the recording to your production size. One of the reasons you might need to do this is because light fields, especially in the raw, take up a lot of data. Like a whole lot. Lytro is currently mum as to the exact number, but it is safe to assume it is many terabytes per hour.
Data like that requires a special solution to handle it, which is exactly why Lytro Immerge comes tethered to its own specially-designed, dedicated server.
“The camera is useless without the server,” says Braunstein. Strong words for a camera manufacturer, but pushing around 4D light field data is neither trivial nor standard. The custom ‘server’ is basically a huge recording cabinet, fully optimized for speed of writing to handle the firehose of data the camera is sending at it.
Lytro advertises Lytro Immerge as being full-stack, from camera to storage to post-production. Meaning that beyond the camera Lytro has built an entire suite of light field-specific video tools as well. The system will integrate with programs like Nuke and other industry-standard software at launch and the integration should have some benefits for today’s VFX artists.
“Light fields allow you to much more easily insert CG into the scene,” notes Braunstein, “and the future of AR and VR is the blending of CG and live action content.” Briefly, light fields allow you to determine the depth of objects in a scene, which allows you to composite and alter raw footage much, much more easily. (We’ll have many more details about how it integrates in a future, technical article. Stay tuned.)
After you create a light field video, you probably are going to want something to watch it back on. Lytro has built a player that will leverage the entire 4D light field, allowing positional tracking, for example, during playback. Initially, the company is only focused on ‘higher end’ VR headsets like PSVR, the HTC Vive and Oculus Rift, but others are likely to follow (hardware constraints on positional tracking notwithstanding, of course).
Beyond the actual player software, Lytro has also built a streaming server prototype, which should optimize bandwidth for streaming delivery by only delivering the content within the user’s field of view (while streaming other regions at lower resolution). Lytro’s team compares it to Facebook’s streaming 360 video solution, though with the obvious added complication of positional data.
Lytro is planning to make prototypes available to content creators in Q1 2016, just in time for the consumer VR headset releases. According to the company, Lytro Immerge will be available for purchase at a cost that is “in line with other high end systems,” as well as being available for rental.
With the Lytro Immerge we are preparing to enter a whole new realm for filmmaking, especially within the VR space and it is something that has even the biggest VR content companies jumping for joy.
“Everybody is talking about light fields and nobody fully understands the potential yet. Light field technology is probably going to be at the core of most narrative VR,” says Aaron Koblin, Cofounder and CTO of Vrse. “At Vrse, we’re just waiting for the moment when we have the tools. I think both the capture and playback of light fields will be the future of cinematic virtual reality.”
Speaking with Lytro CEO Jason Rosenthal, he says that the company is exploring the idea of creating “multiple rigs,” which we can guess might vary in terms of the size of their capture volumes. Additionally, Rosenthal expressed that the company is exploring some other interesting uses of light field technology that they are not yet ready to discuss publicly. Looking forward, Lytro seems to have a big road map for virtual reality and light field cameras, and Lytro Immerge is only the beginning.
