Back in early fall 2014, I was shown my first virtual reality experience on a Samsung Gear VR developer kit. I was so overwhelmed and excited by the possibilities of what I just seen that I basically dropped everything I was doing in the film business (pretty much on the spot) and launched UNLTD VR, an immersive content and technology company. As a 20+ year film and TV content producer and distributor, my goal was simple: create ground-breaking cinematic immersive content that would compete with anything a Hollywood blockbuster has to offer. What I’ve learned along the way is that creating high-end virtual reality presents several nuanced challenges.
Hardware and software limitations
When we first began creating VR experiences in late 2014, we barely knew how to stitch 360 images together, or how to create perfect stereo in VR. We were working with early GoPro mono 360 camera rigs, or massive RedCam nodal stereo rigs. We were improvising on the go, making it up and innovating as we were shooting. Our background in developing 3D cameras and shooting techniques for the film industry certainly helped as we adapted existing technology to our needs.
On set, we were learning an entirely new way to design, block and shoot a scene in VR. In those first days, we were hiding the crew, equipment and lighting ‘in the stitch”, and then shooting a clean plate after to complete the shot. We had to learn new ways to direct actors and action to accommodate the limitations of multi-camera 360 rigs and the no-go parallax zones between lenses that are still a problem today.
In post, we were doing our best with off-the-shelf stitching software that we quickly learned was not up to the technical spec that we required for high-end VR. We quickly realized that we had to move on to higher-end post platforms such as Nuke and After Effects, though we had to adapt those to our need as well.
As our company grew and we began doing more complex brand projects for clients, we suddenly found ourselves with an on-staff team of very talented FX artists. We also built a medium-scale render farm to handle the massive data required for the processing of high-end 360 images and effects. When we started UNLTD we never imagined we’d be running a VFX Studio, but we had no choice – there was simply no other way to get the job done at the level we wanted to compete at in the VR space.
Creative storytelling challenges
Storytelling in virtual reality is different from any other platform, which is what I love about it. Stories don’t have to be linear and viewpoints can change seamlessly. The storytelling style used differs depending on the type of VR experience and the intended audience. This is especially clear in branded content versus original film content.
For the past 12 months, we have been working on “Trinity,” an interactive VR series we will be releasing the pilot for later this fall. “Trinity” is a live-action sci-fi thriller set 2000 years into the future, in a desolate world where androids and machines reign supreme. After a few years of working almost exclusively on brand VR experiences, we decided to go big on our first fiction content VR project. This series combines every possible challenge you could imagine in a VR production, from technical to creative. We have had to develop new technologies to achieve the desired final product experience.
One of the most important elements we wanted to integrate into the series was the ability for viewers to move through the story. At UNLTD, we believe the era of “fixed” or stationary VR experiences is over – where the viewer is pretty much stuck in one place and the action moves past us. This limitation is due to the nature of most VR cameras and the complexity of “stitching” the multiple live images together into a 360 image. Although many wonderful experiences have been created this way, the future of VR is definitely dynamic. “Move or die” is our new motto.
Post-production issues
However, when you add movement to a live 360 image, the complexity of the post-production process rises exponentially (like really, really exponentially). This happens both in terms of both stitching the images together and then creating visual FX. We started by developing a VR camera specifically designed for movement, then had to adapt new stitching techniques to account for camera movement, and finally changed our on-set direction to minimize post-production work. More than ever, VR production is made or broken on set – certainly in terms of blowing the budget in post. It took a ton of testing before we finally hit the floor for the “Trinity” shoot in Prague last November.
To further enhance the viewer’s ability to move through the story, we decided to “double-shoot” everything, using our own Sputnik Motion-VRcam for all the live action shots, as well as an array of volumetric cameras capturing depth and RGB information on the actors and close-range set. We then scanned the entire location architecture for 3D mapping. Combining all these different assets, we’ve built fully interactive environments where we can match and then switch back and forth seamlessly between the live action-action and volumetric environments.
Although volumetric camera technology is really in its infancy, we’re convinced that all camera capture technology will be volumetric within the next few years. We wanted to be one of the first to integrate volumetric interactive environments into an immersive experience, along with all the added freedom it gives the viewer. We adapted the current limits of volumetric imagery to our story, and came up with really cool and stunning environments for the viewer to experience and interact with.
Putting it all together
Finally, assembling a live-action interactive experience requires an interactive engine to put it all together. However, the current interactive engines on the market are designed for games, and have limited ability when it comes to live-action elements and the kinds of cinematic transitions between image and sound that we require. We’ve been working with Unity game engine and have been developing techniques to create editing and transition effects to make “Trinity” a seamless cinematic experience. The Unity team has been extremely helpful and forward-thinking and has worked with us to develop an interactive/immersive content platform that we plan to build many future feature and series VR experiences on.
We believe that true blockbuster “must see” style experiences will drive the mainstream adoption of VR. If we’re to succeed, VR production studios need to push the creative and budget limits and release content at the highest level, competing with anything that film, television or games have to offer.
