Nonny de la Peña is know by some in the VR industry as the “godmother of VR.” Her work in recent years has been instrumental in defining the current VR landscape, and making waves in the emerging field of immersive journalism. Nonny’s work has inspired Palmer, who was once a budding journalist himself, and led to him actually interning essentially with her during her first appearance at Sundance in 2012, where she released the first ever VR piece at the festival. The experience, Hunger, is held in reverence by many in the industry as a shining example of the potential and power of VR.
During our conversation, Nonny shed some light on how her experiences, both Hunger and Project Syria came to be. She also told us the story of how she and Palmer Luckey began working together, what her first VR experience was like, and what the future holds for immersive journalism.
Will: Nonny, you have been involved in the VR scene for a while now, can you describe some of your history with the medium and how you became the first person to ever show a VR experience at Sundance?
Nonny: I have been a VR obsessive for eight years and have been making some interesting pieces for a while. In 2012, I actually did start what I was calling Immersive Journalism. What we were doing was recording audio at food banks and we recorded a guy who went into a diabetic coma waiting in line for food. I felt like the hunger was really invisible, and I wanted to make visible, point you on scene. I knew virtual audit was the only way I could do that. We did a piece in unity, using motion capture and animation, etc. and basically that piece was brought here to Sundance in 2012 as the first VR piece. My team was pretty amazing and included Palmer Lucky, who at the time was a young kid who was building VR goggles in his garage. And ended up in our lab at USC under Mark Bolas who was running the MXR lab. And Mark, who was credited in an incubator situation, where there was me, and Palmer, John Brenan, Bradley Newman, Ty Fan, Julie Griffo, Evan Sumo. I would say those were the key people hanging around who really push all the code, the distortion, the hardware, the feeling, the sensibility. Everything we needed to make that piece work.
“I wasn’t thinking about games, I was thinking about other kinds of stories we could use for VR.”
I was terrified. We show up with two duct tape pair goggles. Nobody’s ever done this before. And opening night, people take off the goggles and they’re bawling, crying, sobbing. They’re down on the floor trying to hold the seizure victim’s head; they’re trying to comfort him. I was completely astonished. That was the point when Palmer was literally crashing in my hotel room and drove the truck back for me. Nine months later, it was the Kickstarter, and of course now Oculus Rift has really made VR possible for everybody. So I’m really thrilled that it’s gone in this direction. I can tell you, I’d have colleagues that’d stand up and point their finger at me and say, “You can’t do that! That doesn’t work!” because I was trying to bring these deeper narratives into VR. I wasn’t thinking about games, I was thinking about other kinds of stories we could use for VR. I love the gaming community and I love the game work, but it just happened to have a different kind of voice for different stories that I wanted to tell.
Yeah, I really noticed that the power of VR cinema is obviously apparent here. It’s something that truly seems to affect people on a deeper level than standard cinema. I’ve actually spoken with Palmer about this at CES, immersive journalism. What is the sort of ethical consideration for immersive journalism?
Yeah, because throughout this I’ve been sort of one foot in academia, so I’ve written some papers about this very specifically on the issue of not to confuse subjectivity with lack of objectivity, with lack of transparency. So my sense is we experience the world through our bodies as much as we do through our mind, and there’s no reason we shouldn’t experience the story in the same way. Just because we’re offering people that opportunity doesn’t take away the approach that we would use in normal journalism. What are the best practices, how do we stay accurate, how do we convey the stories appropriately? These are old lessons that we learn in journalism that I’m just bringing into a new platform. And I was a correspond for Newsweek magazine, I’ve made a bunch of documentary films, this is a space I’m very comfortable working in, I’m just happen to be somebody who didn’t mind learning how to code.
Walk me through how Project Syria got started.
The way Project Syria happened was the dean of the USC School of Cinematic Arts Elizabeth Daily, brought the head of the world economic forum, Carl Schwab, to my lab a little over a year ago. And he tried out Hunger in Los Angeles, the piece I premiered here at Sundance. He took off the goggles and he turned to me and said, can you get something on Syria ready for our next meeting in like six weeks? I know what the vérité moment is, the kind of content and I need you to experience, and I do that first. But in this case I had the direction, and it was really intensely hard. For example, I contacted a Spanish photographer who I thought had a lot of interesting material we could use, but 20 minutes after, it came over Twitter that he had been kidnapped. I sent a team at the same time we were finding the vérité pieces that we ultimately used, including one specific clip where a bomb hits Sweet Nelepa when a young girl is singing. We then had to get all the archive footage around it of the aftermath to continue the piece.
“We experience the world through our bodies as much as we do through our mind, and there’s no reason we shouldn’t experience the story in the same way”
I also sent a team to the border of Iraq and Syria, and the area where Isis is operating right now, and I couldn’t do that. They collected a lot of material for me that the audio, the textures, the sensations, the setup, all of those were utilized to inform how we built the second half of the piece.
What about live action journalism in VR?
One thing that is different about the way that I do these pieces, the two are going to meet. Live action, you’re going to have your project tango kind of thing and you’re just going to scan a world and turn it into 3D models. The two are going to meet. Right now I work with CGI because I like the idea of being able to go behind your TV set. The cinematic nature of live action is beautiful, but you’re still left with one position. Whereas with Project Syria, the difference is when I put you in those goggles [for the demo Nonny used her own HMD crafted specifically for the experience], you walk around freely. No controllers, nothing, I’m wireless. And I allow you to be there with your fully body on the street where this event transpires, and I think that’s what makes it a transformative experience because you’re really there.
So, throughout your experimentation in the fields, what are some of the lessons that you’ve learned about cinema in VR that you might not have known a few years ago?
I think one of the biggest things that people should know when they’re thinking about building in VR is that it’s a unique platform where the body is along for the ride it is a fully embodied experience. Think about what would happen to your real body if you did things to it in VR. Think about what that experience would be like. Hopefully that would mean we don’t make too much people nauseous, if you can think about what would make you in the real world, it might make you nauseous in the virtual world.
“I think with VR we’re going to have a different kind of brain, ultimately, that’s going to be as comfortable as we are now with the jump cut.”
That doesn’t mean that’s always going to be the case. We know that when somebody uses a hammer a lot, literally their brain is changed by their knowledge of the hammer. Something new happens. When you take violin lessons, something happens to your brain. I think with VR we’re going to have a different kind of brain, ultimately, that’s going to be as comfortable as we are now with the jump cut. When you’re first starting out, really be conscious that this is not film. You can’t have a whip pan. Jump cuts can be weird. Don’t shoot people along spaces. Give respect to the body.
How about in terms of narrative structure? What lessons have you learned about how narratives should be structured in VR?
So I think again, I’m a little bit unique in I do tend to do linear narratives, but they’re spatial which means designing these for the idea people could look anywhere. They can jump, they can run, they can bend, especially in the larger IMAX version. I call it an IMAX like system because it’s an installation at this point. Knowing people can look anywhere is really important to be thinking about how can the story be told and not lose the storyline. And that’s something to take into consideration.
The other thing is when I did hunger back in 2012, I had triggers. The original event occurred over 20 minutes in the course of pieces edited down. The woman who was running the food bank line, she’s really overwhelmed, and she’s yelling at tall the people. And so I edited it down so there’s only a certain amount in the piece, and if you walk up to her, I use some of those phrases and I had her direct them at you. Similarly, when the emergency workers came, they were really terrible to the people in line. And I had the same thing if you walked up to the emergency worker, he would yell at you.
“I built [Hunger] from the heart and that seems to be a little bit simplistic to say, but that piece never would’ve been made if I hadn’t been so damn committed to trying to get other audiences to understand what was happening in this country for the hungry.”
But I ultimately haven’t been doing that lately because I sort of feel like I don’t really want you to change the storyline. In a funny way, I’m returning back to some more and more traditional perspectives on how to create narrative flow. The truth of the matter is this language isn’t written, and there’s a lot of creative minds coming into this field. Which is beginning to see all kinds of aspects and content, and I’m being the last person to say that I know it all. It has been fun for me. People like Dan from Dennis and Chris Milk, they both experience VR for the first time with Hunger in Los Angeles to give you an idea how significant that was. So you get major artists who have become as interested in this field as I have based on that one nutty, crazy piece that’s CGI. I built that piece from the heart and that seems to be a little bit simplistic to say, but that piece never would’ve been made if I hadn’t been so damn committed to trying to get other audiences to understand what was happening in this country for the hungry.
What was your first experience in virtual reality, and what was that like?
I can tell you exactly my first experience with it, really blew my mind. There’s a guy named Mel Slater and his wife Yue Sanchez. Mel is just an old VR Researcher, was there from the very get go. And I was invited into his lab after I had made a virtual Guantanamo bay prison before Hunger. He was working with a researcher in London who had gotten closing footage of the Pub, CCTV footage. And they’re analyzing when do bar fights break up and what do bystanders do. We can’t really put somebody in a fight! Or at a fight. So we created a VR experience, and you put on the goggles, and at this point, it was the wide 5 which was originally designed by Mark Bolas who created the incubator, where Palmer and I and all those other folks I mentioned showed up. Mel had one of Mark Bolas’ wide 5s, and I was standing up, and we’re suddenly in a bar. He had great audio. And this guy walks in and the other guy in the bar stool seat gets off and he’s furious at this guy for having the wrong sports jersey on, and he’s ready to fucking beat the shit out of him.
And it blew my mind! That was the first time—full bodied, I’m walking around, I’m standing up. Oh my god, I can never put my audience to sit out there in seats again. I have to put them onstage, in the story. And that was it. From that moment, I could never look back.
That’s incredible. So you said that you and Palmer were involved in an incubator together? How did you two get connected? I know he has a very high regard for the work that you’ve done in VR so far.
The way thing started is Palmer had the largest private collection of VR goggles. He ended up showing up at the USC institute for creative technologies mix reality lab, run by this guy named Mark Bolas. And mark was best known for creating the goggles called the wide 5, and was leaving this lab and leaving all this research. There was Palmer who was kind of putting himself through school at long beach state. Mark let him into the lab to start hammering away at staff. At the same time, I had a fellowship in the USC school of journalism and research faculty. But my fellowship ended and I had decided to do a doctorate, but it was kind of year before. I had a year in between in which I wasn’t doing anything but be obsessed with VR. Palmer was there and a couple of other people ended up in this, I call it an incubator, but it was sort of this moment where Mark Bolas let us come together and play.
Again, it was John Brendon. A guy named Ty Fan. Bradley Newman did a ton of work on Hunger in Los Angeles. Evan Sumo was doing a lot of the code. Julie Griffo was also a student there. We had a unity working group and we were really teaching each other and working together. Out of that moment, some pretty amazing things happened. People went off and did some crazy wonderful stuff. I credit Mark Bolas for creating the environment that let us all come together and follow our passion of VR.
In the next 5 to 10 years, where do you see VR journalism headed?
Hm… I think you’re going to have a range of experiences. I think you’re going to have your IMAX like theater location base where people will go to see fictional narrative and non-fiction and have experiences with Bradley here. And then you’re going to have your home entertainment experiences, which I think Oculus will be the best at. And then I think you’re going to have your mobile experience. That mobile experience is going to be crucial to the news.
“I guarantee you the way that everybody’s working on scanning, the way everybody’s working with VR and headsets and viewers, we are going to be able to quickly scan up the Boston marathon bombing site or whatever, and people whip out their phone, drop it in, and take a look at what happened.”
I talk about certain stories about how quickly I can generate some of these pieces in 48 hours. I used the Trayvon Martin case, it’s getting a little old, but I should get another sample to show people. It was a rainy night, its cookie cutter condo buildings, it’s one car that could get onto one of the modeling sites for only 125 bucks. It’s only animating a couple of people, it’s dark, it’s rainy. That story I could make you feel like you understood what happen, where it all transpired. Obviously the cabinet knows exactly what happened between the two men at the end, but you’d be able to stand in the different apartment condos with the people who are calling 911. You’d be near them, looking out, in the rain, and understand the whole location. I’d be able to put that piece together in 48 hours. I guarantee you the way that everybody’s working on scanning, the way everybody’s working with VR and headsets and viewers, we are going to be able to quickly scan up the Boston marathon bombing site or whatever, and people whip out their phone, drop it in, and take a look at what happened.
You see that happening at the speed of Twitter?
Not the speed of Twitter, but the speed of YouTube. [laughs]
When it reaches a wider scale and more people are able to see journalistic experiences in that sort of way, how do you think that’s going to change us as a society culturally?
I don’t know. I think of immersive journalism as being another platform. We know that radio changed newspaper, and we know that television broadcast changed radio. I think that virtual reality is going to be the same sort of thing. It’s going to be another platform that will change things, but it’s just going to offer another way for us to understand more deeply and resonate the way we get our stories. My kids are 11 and 13, and they are very comfortable in virtual environments. This is not a big deal to them. It’s a big deal to those of us who are maybe a little bit older than that. For their age group, they’re going to be very comfortable with getting their news this way. And guess what, they aren’t necessarily picking up the newspaper or watching broadcast TV, and we really want to continue to have—good democracies are based on informed citizenry. I mean, when you stand, I see so many people go through Project Syria go, oh, I understand now! And it’s only a minute or two of feeling that they’re actually there that they get all this information that they didn’t understand before when they read about in a newspaper or saw it on TV. I think that’s going to be true for education, for journalism, and for a lot of amazing wonderful narrative pieces that are going to come out beyond even the entertainment and gaming and health and medical.
Nonny, thank you so much. This was a fantastic interview.
