Get that Cardboard viewer out and get it warmed up, VRgonauts! In this final installment (Part 1, Part 2, Part 3) we’ll discover the transformative power of audio (when we could get it to work!) We’ll also tweak our final bugs and present our finished short to the world!
After heated months of head-down production, my animation team and the VR video group at Idle Hands Studio had finally arrived at a version of the short that had a locked visual. That meant no more changes to the editing or the animated visuals. We all agreed that the visual story was working well with the narration, so it was at this point that I had my first conversation with Weston Fonger at Vinyl Mix to talk about overall sound design for the piece. Not many animated pieces have the budget to bring on a dedicated 3D spatial sound designer, but TED Ed was excited about what they’d seen so far and was willing to give our project the extra life that dedicated audio can bring. I was excited, too, and was bubbling over with ideas about how we could place sounds at specific locations within the 360 sphere, just as we had done with visuals.
My first conversations with Weston centered on the vision that the environmental audio and the sound effects needn’t necessarily be literal recreations of what was being viewed. I’d hoped that a more “symbolic” sfx track might give the audio a larger presence, in the same way that the visuals weren’t slavishly following the narration track. Note in particular when our Shaman character flops down in his chanting position. We’re hearing a ringing prayer bell and a woodwind-like reed instrument that sort of imitates his breathing and chanting, but not specifically so. Also listen for the offscreen chorus as Maria first discovers the galleries in the cave. Their solemn chanting adds just the right touch of awe to the opening audio overture. This is Weston’s handiwork.
Concurrent with my sound design conversations, I brought in composer Stephen LaRosa to add an “analogue-prehistoric” musical score that would compliment the sound design but not overpower it. This was a tricky proposition as these two audio specialists had to be aware of each other’s output and do a delicate dance of knowing when to come on a bit more forcefully in the mix and when to lay back and let the other teammate take the reigns for a particular section. Fortunately Stephen and Weston had a history of working together and their intricate puzzle of interweaving the respective audio specialties was presented in a delightful mosaic of sound. You’ll note how the music in particular ebbs and flows throughout the piece, allowing the narration and specific sfx hits to take precedence. Or perhaps you won’t notice it. In some ways, a seamless audio experience is most successful when the viewer isn’t acutely aware of everything they’re hearing. Rather, the audio is woven into the entire story tapestry as an integrated element. That’s when, as a director, you know you’re working with very talented craftspeople.
So you may be thinking at this point that it was a smooth and easy coast to the finish line. But you would be very wrong. In those last weeks I discovered that 3D spatial audio was not a fully implemented technology across all platforms. I first became aware of our audio issues when I attempted to critique our first sound mix, and I was unable to get the spatial audio to work correctly. In fact, I got no spatialized audio at all. I was just hearing a stereo mix. When I alerted Weston to this he replied, “Oh, I’ll bet you’re using an iPhone, right? I forgot to tell you that the iOS YouTube app doesn’t wholeheartedly support spatial audio”. I was flabbergasted! To be so close to the finish line on this project, only to discover that there was a whole world of new technical hangups to overcome. But as I’d grown accustomed to doing when confronted with the many tech hurdles on this piece, I took a long walk and a few deep breaths, and plotted our path for getting the piece over the finish line.
Weston presented me with a few alternatives. But none of them included presenting the piece in 3D audio for all users. It wasn’t his fault. The YouTube app wasn’t supporting all versions of the iOS software. Just as I was about to pull the trigger on the least objectionable audio option, I tried one last desperate effort. I updated all of my phone software, and when I had finished, unbelievably, the audio worked! It seemed that in the few weeks that we had been attempting to implement the spatial audio, YouTube had updated their app! Now it was miraculously running on the latest iOS with no problems and in full 3D spatial mode. Hallelujah. It looked like that elusive finish line was finally crossed and the piece was officially “done”.
So that is the story of our work and how a new genre of entertainment was birthed by a dedicated crew of talented creatives who refused to quit. Our hearty thanks go out to the TED Ed Executive Production Team for their tireless support throughout all of the peaks and valleys inherent in creating a trailblazing piece like this, which, even a month ago, would have been impossible to complete as successfully. That’s how close to the edge we were working, and now it’s time to unveil our handiwork. Wanna see it?
To see the final piece go to https://youtu.be/CX4KuIBmnjI . It can be viewed on your desktop using your mouse to “push” the video around in a 3D space, but watching on your phone through a Cardboard viewer with headphones is really the ONLY way to experience it as intended.
You will also want to check out our helpful Viewing Tips below to ensure that you’re set up to enjoy the piece in its highest resolution.
Viewing Tips for Google Cardboard
- Procure a “Google Cardboard” type viewer for your smartphone.
- Attach headphones to your phone, and adjust the volume.
- Navigate to the VR short’s URL through the YouTube phone app.
- Confirm that you are watching via the YouTube app and NOT through your phone’s browser. If you are watching in the app you will see a small “cardboard viewer” icon on the bottom row of the video window. Hit “pause” on the video.
- Tap the 3 vertical dots on the top right corner of the view window. This will slide up a submenu where you will choose the quality setting of your video stream. Choose “2160s”. Note that if you are not streaming via wifi YouTube will ONLY allow you “720s” quality.
- After choosing your quality setting, tap on the “cardboard viewer” icon on the bottom row of the video window. This will present the video full screen in prep for the Cardboard viewer.
- The screen is now divided into 2 halves, separated by a thin white line that runs halfway up the screen. Make sure to rotate your phone so that this thin line is coming from the bottom of the screen. This ensures proper stereoscopic depth.
- You’re Done! Insert your phone into the Cardboard viewer and press the viewer’s button to hit the “play” button the phone screen. The video will begin.
Thanks for reading along! Drop me a line and let me know what you thought of the piece. A hearty “thank you” to our innovative production partners at Idle Hands Studios, and to the Executive Production team at TED Ed for their unwavering support of this project.
Michael “Lippy” Lipman is a classically trained 2D animation director who first found success as a feature film animator in Hollywood. With the introduction of interactive media he transitioned to producing major entertainment pieces for CD ROMs, console gaming, online advertising, and internet-based animation. Currently his company 360360VR is enlisting VR/AR technologies to further the immersive storytelling possibilities for his client, TED Ed and others. He lives in the San Francisco Bay Area.
