As of today, anyone who has a Gear VR headset can download a Netflix app, allowing a wide variety of content to stream into a virtual environment on a Samsung phone. The announcement made at Oculus Connect 2 sparked a ton of articles from just about every major tech publication touting VR’s mainstream adoption. Shortly after the breaking news, Netflix published a post with the technical problems they solved during the development process.
Over the course of a month, John Carmack, CTO at Oculus, worked closely with engineers from Netflix. As Carmack mentions in the post, “the plan of attack was to take the Netflix TV codebase and present it on a virtual TV screen in VR. Ideally, the Netflix code would be getting events and drawing surfaces, not even really aware that it wasn’t showing up on a normal 2D screen.” He then created a ‘VR 2D shell’ that functioned like a simplified version of the Oculus Cinema application. Because of this, the big screen is rendered with Oculus’s peak-quality TimeWarp layer support, and the environment gets a dynamic lighting effect based on the screen contents.
The core Netflix application uses two Android Surfaces – one for the user interface layer, and one for the decoded video layer. To provide smooth playback, video frames are queued a half second ahead, and tagged with a “release time” that the Android window compositor will use to pick the best frame each update. Carmack modified the Netflix video decoding system so it would call to the VR code after it submitting each frame. This lets the program know something with a particular release time was sent. It then immediately updates the surface texture and copy it out to a frame queue, storing the release time with it. This is an unfortunate waste of memory, since it duplicates over a dozen video frames that are also being buffered on the surface, but it gives timing control, which is much needed for streaming content.
To get the display working properly, the resolution was reduced to 720p HD, which as Carmack describes in the post, “is the highest resolution video you should even consider playing in a VR headset today.” This has to due with content protection as studios insist on taking steps to secure the execution environment of HD content. Alternatively, reducing the video file sizes helps with streaming capabilities as well.
Finally, Carmack addressed power consumption problems associated with watching long-form content. At first, when the initial system was loaded, the phone overheated to the thermal limit in less than 20 minutes. Not good for VR. To reduce heating issues while accounting for minimal losses in quality, “Asynchronous Time Warp” was implemented. Head motion is what takes a heavy strain on the system, so cutting out the virtual environment keeps power consumption low. Because of this viewers can enter a “void theater”, where everything is black except the video screen, which obviously saves additional power.
On a side note, it looks like the Gear VR Netflix out will come with a “Travel Mode” as well. This was uncovered in a Road to VR video showing the UI. We wrote about using VR on a plane a few months back which brought up several issues with drifting. Hopefully this feature solves those problems.
