Every technology needs a corresponding business model to thrive. Facebook’s technology is a social matrix which intelligently channels user generated content interspersed with advertising through a massive social network. Powering the model requires billions of living, active users. For virtual reality to become a core technology for the social matrix it needs billions of users. Facebook has said as much, placing the technical responsibility of jacking one billion users into virtual reality – and with it the social matrix business model – in the hands of the same engineer who went to hell and back again to bring us 3D gaming: John Carmack.
Myth Becomes Legend
Oculus Connect 4 featured John Carmack’s fabled unscripted talk, a tradition which began in his days at ID software during the annual gathering of the faithful – Quakecon. I was there, 15 years ago as Carmack preached to an audience of mostly gamers as long as the powers-that-be would let him go on, or until his voice gave out. He was and remains the paradigmatic software engineer, wielding code to whatever end; today, a comprehensive platform for mobile virtual reality, and in the good old days, killing zombies. This year Carmack’s talk was an advanced class on the challenges of squeezing virtual reality onto mobile devices, refining solutions and optimizing for stand alone systems, now that many of the pressing and foundational issues have been solved.
Carmack relentlessly and methodically uses the tools at his disposal to solve challenges spanning the entire mobile VR stack. He is focused on delivering tangible solutions on today’s hardware, and the code which ships serves as a foundation for the Oculus development community. The future makes for interesting conversation where “reasonable people can have radically different opinions.” He has learned the necessity and liberation of leaving that to researchers like Dr. Michael Abrash who helped guide technology development at ID and now Oculus, or Dr. Bruce Naylor who conceived the “BSP Tree” mathematics that powered the DOOM and Quake 3D rendering engines.
The Mobile Gap
At Tirias Research we have forecasted a perpetual 10 year performance gap between mobile and desktop VR systems. As a corollary, John warns in his OC4 keynote that while Moore’s Law is likely to yield an order of magnitude increase in performance, he believes it was unlikely we will see “a second order of magnitude.” Mobile VR will achieve the performance of today’s desktops – at today’s Oculus-recommended specification – in about a decade. In the period that follows, we are likely to see a radical slowing. Given that truly immersive VR requires higher resolutions and improved visual processing, the challenge of driving the pixel count and advanced rendering expected for future displays would crush today’s desktops and will leave the mobile VR systems of the future in the dust.
It is hard to give a full and fair assessment of the mobile VR ecosystem in this story but I’ll summarize the sentiments I’ve heard from developers. Many are optimistic about the level of immersion which might be squeezed out of a modern, mobile GPU. But the majority are are concerned. Mobile VR, whether due to its performance and power limitations or the still-nascent art of making VR fun, is not producing the experiences necessary to monetize content, much less to deliver a billion monetizable users (with one or two exceptions). Countless conversations yield the same results – mobile is its own platform requiring dedicated effort. However, the inescapable truth remains: performance and power limitations stand in the way of delivering richly appointed user experiences – the visual and experience realism required to make VR almost magical.
Today’s mobile developers are learning to stretch mobile VR – Mr. Carmack at the platform level – and the developer community at the experience level. The upcoming Oculus “Santa Cruz” and “Go” platforms rely on today’s mobile processors, or if we are lucky with Santa Cruz, maybe one generation beyond. Right now, the performance of mobile VR for 2018 holds no secrets, these experiences can be designed and run today at the performance anticipated by these upcoming stand alone designs. Some developers argue that it’s the hardware, while many, like Mr. Carmack, argue it’s mostly the software. However, none argue that mobile VR is hard, and so far, something hasn’t “clicked”.
The Red Pill
Unspoken at Oculus Connect – but heard in industry whispers – is the use of the cloud to stream VR experiences to the user. Shifting the lion’s share of the processing problem from headset/phone to a server could allow contemporary mobile devices to deliver high-fidelity virtual reality worlds – complete with the user base required to power the social matrix rendered in virtual reality. Recent advances in video compression, virtualization, networking, and real-time rendering have evolved since the days of Onlive, an early-cloud based game platform. These technologies have advanced significantly since then, and appear poised to tackle VR streaming. Perennial innovators like OTOY are rendering photorealistic lightfield scenes in the cloud and rendering at high resolution on mobile devices. Foveated rendering is a set of combined technologies which can reduce VR’s computational load by using eye tracking and advanced rendering to focus on only the highest priority pixels – those in the immediate gaze of the user. At the same time, video compression technologies continue to be refined – squeezing more visual fidelity into a given amount of network bandwidth. The race to develop novel approaches for delivering tomorrow’s streaming VR content is running hot as innovators and academics approach solutions aimed at solving the very difficult and critical issues of motion-to-photon latency and mobile network bandwidth limitations.
Today’s processing giants – companies like Qualcomm, AMD, and Intel – are also taking note, demonstrating advanced wireless technology, specifically targeting its application in VR. The implication is not that we will simply have untethered VR headsets, but that headsets can be virtually any mobile screen, powered by a future, server-based VR infrastructure and an entirely new model for VR content processing and delivery.
There are three real challenges in the use of the cloud for interactive virtual reality. Together they must hit the magical sub-20ns “motion-to-photon” latency target necessary to achieve presence and avert simulation sickness.
- The Wireless Network: The requirement for high bandwidth/low latency connectivity will stress wireless networks but also drive their development. The convergence of next-gen WiFi and cellular wireless technologies (likely in the form of WiGig and ultimately 5G and beyond) will require widely deployed, unified solutions for enabling low latency, video encoded, prioritized VR stream delivery.
- The Infrastructure: Cloud-scale software and hardware must meet VR performance requirements, dynamically allocating CPU, GPU, video encoding compute and network resources. Networks must be configured to minimize latency through proximity of servers to customers, and optimized network traffic routing, and edge computing.
- The Distributed Render Engine: Game engines designed to take advantage of cloud scale hardware, rendering from the cloud to video streams with connectors to critical I/O metadata upstream and downstream over the network. Rendering technology including light fields, eye tracking & foveated rendering, and local processing must be combined to deliver user experiences on time and compensate for inevitable delays in the delivery of network traffic.
Change is born from innovation, and great, disruptive change comes from the intersection of multiple innovations. Today’s work on mobile optimization of VR is helping to improve computational efficiency and user experiences. In the future, this effort will evolve into efficient server processes which run in the cloud, streaming with high levels of performance to efficient and affordable mobile headsets. Headsets will gather head tracking data on the front end, and just prior to rendering synchronize cloud streams with up-to-the-millisecond data on head position to ensure the user has the best possible visual experience. Developers will be able to worry less about the compute performance limitations of mobile devices, and more about creating compelling VR experiences. Developers will benefit from an end-to-end server to client architecture that ensures experiences are delivered as designed. The question is, when and not if, the cloud based VR platform will pass out of research and become daily work for Carmack and those that follow. When it does, we will be on a path to a billion users in immersive, expansive and social virtual reality.
—
Simon Solotko is an accomplished VR and AR pioneer who manages the AR/VR Service for Tirias Research, serves as a mentor at the German Accelerator and adviser to technology innovators. He has been attending John Carmack’s unscripted talks off and on for almost 20 years. You can follow or message him on Twitter at @solotko using twitter or connect on LinkedIn. To read more on the migration of VR to the cloud and the intersection between VR and AI, check out http://www.tiriasresearch.com/pages/arvr/.
This is a guest post not produced by the UploadVR staff. No compensation was exchanged for the creation of this content.
