There’s one voice in the darkness, a single singer solemnly making declarations about music and the court of song. Other voices come in, and then the same singer is there in five places around me, harmonizing. And he does seem there, song enveloping me.
Hallelujah is the latest video from Within, an acapella rendition of Leonard Cohen’s famed tune. It is a sacredly soft version, only performed by one man, Bobby Halvorson. But in contrast, it is boldly presented, as the video is a partnership with Lytro, and uses their lightfield camera for VR to capture the singer and the church he later stands in.
https://vimeo.com/206480546
“To get to really great VR, you need a whole new set of tools and technology to make the content,” said Jason Rosenthal, the CEO of Lytro. “Hallelujah is the first example of what things might be able to look like when you’re doing true volumetric capture.”
The promise of Lytro’s camera is to not flatly capture the image of a place, but all of the spatial qualities a place. What does that mean? Well, unlike most of the live-action VR videos at Tribeca this year, it isn’t just 360-degree video. Nor is it simple 3D. It is a volume with a full six-degrees of freedom, more like the animated VR shorts, but not generated by artists at a computer. It is our live-action world recorded as data and then all that data processed into a volume. Facebook has shown a new interest in the very same technology this week.
Besides the sense of freedom, Lytro’s lightfield camera and its 95-lenses records the reflections and highlights of light off surfaces to also give the space a more realistic feel, and not a flat animated photograph feel. These are realistic places you can explore in your Oculus Rift or HTC Vive.
Rosenthal said, “If you look at what’s being done in VR today, you have this Sophie’s Choice. You can either get immersive stuff by building on top of Unity or Unreal, but you’re stuck in a videogame world. Or you can get photoreal capture of live subjects and live environments, but you’re stuck with the tyranny of 360 video. What we’re trying to do is bring the best of both.”
This all comes together in Hallelujah. At the start of the video, as we look upon the singer, we can lean to the side and see the volume of his body. It feels like he is there, in a crisp reproduction reciting Cohen’s beautiful lines.
https://vimeo.com/213266879
Though at first he seems to be floating unrealistically in the darkness, when the background fills with the image of St. Ignatius church in San Francisco, the grandeur of the space is felt, enhancing the religious overtones of the tune. Perhaps the imagery is a bit obvious for a song named Hallelujah, but it is effective.
In another video from Lytro, a demo the company made almost a year ago called Moon, we see the harsh lighting of a Hollywood set on an artificial moon surface. That use of light wouldn’t be possible if not for the light-capture camera.
“In the real world, as you move, your perception how the light is moving changes and typically in VR it doesn’t do that. And being able to represent how light scatters and that is positionally dependent and you want to play that back in VR,” said Rosenthal.
Compared to the previous iteration of the camera used on Moon, the space in Hallelujah is larger, the viewer able to move in 6 DoF more. But the tracking volume is still limited compared to other VR experiences made using graphic engines like Unity or Unreal. But as the jump between Moon and Hallelujah attests, the cameras will continue to improve, as well as other parts of the company’s Immerge system for VR,
The promise of Lytro’s cameras can’t be downplayed. Many live-action VR videos have an image that unrealistically bubbles, or the stitching of the image is noticeable, or the convergence of the 3D is not quite right. It feels like the world bottled in a gimmicky way, shoehorned into a new medium. The images of Lytro feel more like the real world replicated. Maybe not 100 percent, but more than other technologies being used right now.
The entire Immerge system that does all of this is more complicated than typical camera work though: the great hexagonal camera with its 95 apertures records the light of a space. They had to this five times, using five different angles, to catch everything of St. Ignatius. Then a batch of servers work together to crunch the data into an actual image. Productions companies have to use a special editor instead of video editing, and then a special player is needed to properly playback all this spatial light data, but the player software will be embedded into the Within app for the launch of Hallelujah.
And this is all done with the future in mind. Though current headsets are only about 2K resolution per eye, the company is currently recording the data of a space at 10K per eye that can be later processed into a 6K per eye image. The data can stay in storage until it needs to be rendered into a higher-resolution for future VR headsets with better resolutions that the first generation out now.
“When we think about going forward, we are way higher resolution and way higher dynamic range than any headset can display today,” Rosenthal said. “Our creators want to futureproof their content. They want to capture it as the best they can today with the knowledge that VR is going to get better and better.”
Lytro will work with others to get 5 to 10 projects launched this year, to showcase what the technology can do for VR filmmakers. And to show the creativity diversity possible with Immerge. But even further out, Rosenthal sees VR evolving beyond film.
“If you look at VR today, you get games on the one side and narrative experiences over there,” Rosenthal said. “There will be game experiences that only make sense for VR. But the bigger thing that will happen is that there will be some form of VR entertainment that we haven’t seen yet that borrows some from narrative and some from the interactive world. That will be the breakthrough experience. We want to be part of making that vision happen.”
Editor’s Note: The piece originally stated that Hallelujah was shot at St. Ignatius church in Los Angeles. St. Ignatius church is actually located in San Francisco on the campus of USF. We apologize for the error.
