Virtual reality cinema is coming at us like a freight train, and with it comes a whole new medium for filmmakers to explore. With the announcement yesterday of Oculus’ Story Studio, it is clear that cinematic experiences are going to be a huge part of the VR content ecosystem, and until an input solution is found will likely have a more immediate impact than games.
But filming in VR is a whole new ballgame for filmmakers and that means new rules to learn. So I turned to IanHunter, director of Kaiju Fury, a newly released VR short cinematic experience to shed some light on the situation. Hunter is the co-founder of the illustrious effects studio, New Deal Studios, and has worked on a number of high profile films including The Dark Knight and Interstellar, for which he recently received an Academy Award nomination. His 25 years of experience in the business lead him to constantly seek new challenges, which is what has led him to VR.
Hunter had a lot of great advice for future VR auteurs, including the five biggest lessons he has learned through his experience with VR filmmaking, and his perspective on the “rules” of VR cinema, if even there are any.
Will Mason: How’s Sundance been treating you?
Ian Hunter: It was a bit of a surprise to us actually. We put Kaiju Fury together, we were experimenting with VR and trying to figure out how to make narrative VR work. We started showing it around and Sundance contacted us and invited us to come out. So we are pretty happy to be here!
In filming Kaiju Fury I am sure you had some technical hurdles you had to hurdle initially. What are some of the best practices for camera setups that you have found in VR?
In the case of Kaiju Fury I come from a company called new deal studios in Los Angeles, dealing with visual effects photography, digital effects photography and production photography. So we are pretty experienced with the various camera set ups available. When the opportunity came along to do a VR project, we jumped on the chance because it was another different way of telling a story and different techniques to use. We started with the Jaunt camera, which is a 3D 360 camera system using multiple cameras to create live VR content. Since we wanted to push things a little bit we used miniatures, so we actually built a little city and monster suits and filmed the destruction of the city. It’s a cautionary tale of man messing with the laws of nature and the consequences of what happens [laughs].
“The basic rule is there is no rule. A first person long take VR experience is as valid as a quick cutting, location changing, changing scale where you go from live action to miniature VR experience.”
We placed the camera in the middle of the action and in this particular system there is a range that you need to keep the action away from the cameras, you can’t really get super close up or the 3D needs to be worked on. So to make it work automatically, we had to stay at a certain distance. We were told that you can’t move the camera, you can’t cut, you always have to use first person narrative, using the camera as a person. And one of the first things that we did was say well we are going to move the camera, we are going to use cuts, and the camera is going to be more of an observer, not necessarily someone who is actively involved in the scene. Because these are sort of conventional film rules that have worked for years and we thought we could apply them to VR. The basic rule is there is no rule. A first person long take VR experience is as valid as a quick cutting, location changing, changing scale where you go from live action to miniature VR experience. I think VR is a medium and it is up to the storytellers to manipulate it to tell their point of view.
When you’re filming a traditional film with a traditional camera, you have a film crew that resides behind the camera. But in 360 VR video, you cant really do that. So what are the best practices for hiding a film crew for a 360 movie?
In Kaiju for example because of 360 VR, there is no behind the camera. So in that case we had to go into what we call previz, where we actually previsualized the set and the camera angles on the computer and then had to check our views going all the way around to make sure we had a set covered every single direction. Then we designed a set that we could see from forwards, backwards, left and right and above us. We decided to do Kaju as a night scene so that hides a lot of sins. We used a lot of smoke, because when you use miniatures smoke helps to increase the scale things appear to be further away when they actually aren’t.
“Its much more like theater, I call it ‘theater of the eye,’ instead of theater in the round.”
With lighting, you can’t exactly hide lights, so we actually built a lot of the lighting into the set itself. The buildings had lit windows, the smoke itself, when that gets lit adds exposure. We did a bunch of light that was behind the buildings that up-lit and exposed that smoke moreso. Everything should be thought of in terms of trying to work in practical lights as much as possible to eliminate your set and use natural light as much as possible. Also using a lot of rehearsing, that is one of the key points of doing a live VR narrative. You have to rehearse, rehearse, rehearse because you have to get your performers to do it like an engine. They have to be automatic, they have to stay in character. Because once you start rolling, there is no crew, there is no director to direct them, no ED to give them queues and signs. They have to have their characters and pull off their role. So its much more like theater, I call it “theater of the eye,” instead of theater in the round. You are looking from the middle out, so you have to get actors who are great at inhabiting their role, who know how to do business, these are all really important things. So rehearsal, preplanning, finding ways natural ways to integrate your lighting, these are all key parts of trying to pull off a successful live action VR narrative content.
When you have a first person POV film, who is the actor? Is it the camera and the person behind it, or is it the viewer?
That’s one of the reasons why we didn’t want to do all of our content in first person. We were involved in Black Mass, where in that case the viewer is the principal. In a first person approach, there is no actor, you the viewer are the one subjected to the plot.
What about experiences, like “The Party,” where there is dialogue coming from the character you are portraying?
That’s fascinating, I guess in that case you are symbiotically attached to the character, so you are not as subjectively involved but you can’t extricate yourself from that experience. So that’s an interesting thing, that’s what we found with Black Mass, people addressed the character, and that character could not respond. So Greg [director of Black Mass] had to create a device in the film that made that make sense so that when the viewer is in that experience so there is violation of that rule – even though there are no rules – that that character cannot speak. That was a case in which there was a first person point of view, in some of the other stuff we have done including The Mission and Kaiju Fury we changed the point of view of the viewer to being an observer. I think how you tell a VR story if it’s first person you have to really really accound for how you address you as a viewer, because then you as a viewer are really locked into what you are observing, but you as an observer you have a little more freedom maybe to experience what is happening. But that all depends on what kind of story you are trying to tell. In that experience you mentioned, you could put an observers perspective in there and watch both parties from both sides and get a completely different experience and a whole different point of view.
What part of the workflow and other basic filmmaking logistics are really affected by filming in virtual reality?
Well we are processing huge files because we are taking 16 images and stitching them together to create one, and we have to account for the left and right eye for 3D, so the files are large. We didn’t have any lead on how to do visual effects if necessary, so we applied what we knew from past work to visual effects and found that you could both apply 2D effects, corrections, and fixes. Painting out rods, painting out wires, painting our the errant crew member who shows up when he wasn’t supposed to. But also what we found was great was we could then take the complete image and put that into a 3D environment, and start adding additional 3D elements that weren’t there in the original shoot, CG elements. In the case of Kaiju, we added a tank, and moved some other materials. That ability to remain flexible using a 2D or 3D approach to effects work meant that any limitation we may have had during the shoot, we can come back later knowing we can add that as a virtual element, as a digital element. So that was helpful. The other thing of course is editing, which all works fine. But what you want to think about is how long you want the scene or shot to last, because in traditional films where your concentration is just directly in front of you, whenever you cut in and out of a VR experience you want the audience to get oriented to the environment, which often requires longer takes, not having action that happens immediately within the take, time to get acclimated to the space. However, we do find that, lets say there is one environment where we cut away from it and then come back to it we don’t have to make that buffer so long we can actually make it shorter because your mind is already somewhat oriented. So the action can happen progressively faster. So these are things that we are just finding out as we are going along.
What are the five most important lessons you have learned for narrative structuring, filming, etc.?
First and foremost, have a clear idea of what the finished project is going to look like in your mind. Unlike other filmmaking where you can shoot lots of coverage, edit it together and hope that it all works. VR is a raw form right now that doesn’t give you that kind of flexibility and doesn’t give you that kind of mercy. So have a really clear vision before you start, and use that vision to plan as much as possible. So vision, planning. Have a clear understanding of how you are going to create the images you will be using. Because you see so much, you cant take anything for granted as to what is going to be visible in the experience so you have to account for that in planning. From that planning comes rehearsing. That rehearsing allows your crew to understand what they’re doing and what is vital to getting those images, and it becomes second nature for them. So that’s really important.
“There are no set rules yet, so it is not our place to lock those rules right now, its our place to inspire people to try new and harder things and break the system.”
Back to the clear vision, having an idea of how you are going to cut the final piece together which leads to perhaps previzing even storyboarding, as conventional as that is. It is hard to storyboard for VR, but if you can previz in a 360 world and do an initial cut that will fill you in for everything that you couldn’t imagine on your own with step one. Step four is that previz to see where your problems are that you haven’t anticipated and know how to correct those problems. And last but not least, for now try to push it. Try to move, try to cut, try to come up with great transitions, try to mix first person and observer, just right now the medium is not established. There are no set rules yet, so it is not our place to lock those rules right now, its our place to inspire people to try new and harder things and break the system. I know it’s not exactly a best practice, but it is my way of saying the only way we are going to figure out how to use this medium is to push it and that is really the practice we should be applying.
So how are you going to push it next?
We have been thinking about can we do chapters, can we create stories that have multiple avenues or routes that you can take. So that the viewer can have a little more say as to what they are seeing.
So, choose your own adventures essentially?
Exactly. It’s interesting though because I am still a director. Even though you can look around, we are still using light and sound cues to direct you through the story. How much of that is applied, how much of that on our part to make you feel like you are choosing your own adventure, and how much it is on you guiding your own adventure. And that is where there’s a lot of opportunity to create content that has more opportunity for the viewer to be more participatory in their experience.
