You have been invited to join a movement. A rebel cause to make the future a stranger, freer, more interesting place. A place where people, applications, games, all intersect and intermingle seamlessly around us in unlimited configurations. Where the barriers between VR and AR dissolve and our virtual worlds and augmentations can seamlessly intermingle. A place without central ownership or one-size-fits-all rules for existence.

To get there, all we need to do is remove the barriers that exist between Augmented Reality and Virtual Reality and let them become one thing.

Come again?

Augmenting Any Reality

Check out this video:

What you’re seeing here is Pokemon Go’s AR mode augmenting Skyrim VR. Augmented Virtual Reality. This isn’t some crazy hack with kitbashing and emulation. We simply put the iPhone’s camera up to a VR headset’s lens and tricked the phone into thinking Skyrim was reality. This works for any ARKit and ARCore app today in any virtual reality.

AR Apps augment whatever reality is given to them. They don’t care whether they’re augmenting real or virtual reality.

Every single AR app made today should run great by augmenting virtual worlds instead of the real one (with possibly the exception of AR navigation apps, but even those have a use in VR — they just need to be provided different data).

Why Do This?

About half of the people we tell this to are already on board. The other half ask “Why?”. Why would you augment a virtual world? What does that do for you?

Here are some anecdotal answers:

1. I want to climb up a mountain in Skyrim and find a Pokemon Gym on top that I can conquer by calling up friends and having them also launch Skyrim and climb that mountain so we can take it down together. The augmentation adds a new social motivation to do something in the virtual world.
2. I want to bring my virtual pet with me when I explore new worlds.
3. I can invite my friends to join me as augmentations and give them a tour of my house in Minecraft.
4. I can add my own body or parts of my house into any VR experience as an augmentation.

Here’s my less anecdotal answer:

1. Any value from augmenting the real world carries over to virtual ones.
2. Right now there are millions of VR headsets ready to be augmented. That’s a market much larger than AR headsets today. Even phone AR apps can run in virtual worlds.
3. It’s easier to augment virtual worlds because virtual worlds are already software, and we can get direct access to game information. In the real world we have to painfully, expensively, and often erroneously reconstruct information from camera pixels and other sensors. This means the quality of augmentations in VR will always be better than augmentations in the real world.
4. When augmenting virtual worlds, we can give our augmentation real power and agency to interact and make changes to the world. Each virtual world can choose how much access the augmentations should have. My virtual cat can fight alongside me in Gorn!

Point 4 above is the really exciting one for me. Current AR is cosmetic only and lacks any agency. I like to think of it like VR without hand controllers. Without hands, you can look at VR and react to it, but you can’t touch it. You are merely an observer. When you augment the real world, your augmentations can look at and eventually observe the real world, but they will never be able to touch it. When you bring your augmentations into VR, it’s like giving hands to your augmentations. Augmentations in VR can be more than just cosmetic!

How does this Protect Against Central Ownership?

I promised we were making the future a stranger, freer place without central ownership. How does Augmenting Virtual Reality do that?

Well, this year major platforms started adopting the OpenXR standard for both VR and AR apps. Runtimes that implement OpenXR have the opportunity to support overlays. I believe Unity and Unreal Engine can both eventually provide 1-click solutions to package your existing AR apps as a native VR overlay. This would dramatically increase the addressable market with AR apps.

This means that, soon, VR apps could start providing overlay interaction APIs to give agency to your augmentations. All AR and VR experiences can be split into environments and augmentations that all share a collection of common APIs for interaction. Note that one possible background for augmentation is the real world. Another is the Skyrim world. And the “metaverse” is just that collection of cross-compatible backgrounds and augmentations.. When we join a friend for a game, we pick a background to meet up in (whether a spot on earth, or a virtual environment) and bring the augmentations that make sense there. Sometimes the background we meet up in will be the real world, and the augmentations we bring will likely be cosmetic only, at least most of the time.

No one owns these interaction APIs, and so no one company will own everything. There’s no one-size-fits-all metaverse, and instead we all get to build whatever world we each want in a socially compatible way. There is no central ownership where an insane man can arbitrarily give control to whatever kid knows the most obscure ’80s references.

The OpenXR standard announced at GDC is an incredible step toward making this weird future come true. It provides a solid foundation to start building these sorts of experiences now. But the standard is still being created and we can all help guide it together to ensure a future we want to live in!

Call to Action

Platform holders such as Oculus, Valve, Google, Microsoft, and Magic Leap need to implement the OpenXR standard, especially the part about overlays and RGB-depth information.  These extensions are the foundation that allows us to augment our virtual worlds.

With Unreal Engine and Unity, we don’t just want to augment virtual worlds – we want it to be easy. Those teams already have (or are working on) a unified framework for making AR apps independent of underlying AR technology. Extend that framework to also give developers that same information in VR for overlays! It really can be as easy for developerssas switching their target to “OpenXR Overlay on PC” instead of “ARCore Android App”!

Niantic should create a PC or VR overlay that lets me catch Pokemon in Skyrim. You should be able to do this without Bethesda’s permission since augmented overlays are already a protected form of art (see Rifftrax for an example where they augment movies with an audio overlay without permission).

Game Developers can then create augmented layers as a cool way to connect all sorts of video game worlds, not just Pokemon and Skyrim. I single out Pokemon Go here but, you could also find Breath of the Wild korok seeds in God of War, Minecraft diamonds in Fortnite, etc.

Some overlay applications for VR already exist such as OVRDrop and Pluto VR. So AR and VR developers should take the time to consider whether your app can be broken up into a background and one or more augmentations. Take my app SculptrVR for example: I now think of it as a sculpting augmentation with a simple skybox background. As soon as possible, I will package it as an OpenXR overlay so that people can sculpt in Skyrim.

This is how we immunize the future from central control and one-size-fits-all rules for existence. Let’s build a freer and more interesting future together!

#AugmentAnything

About the authors: Nathan Rowe is the founder of SculptrVR. SculptrVR is multi-platform, multi-player, multi-scale volumetric sculpting application with 10,000x zoom and also rockets..? Nathan does all of the voxel engine code for SculptrVR, and his dreams are full of octrees. Jared Cheshier is the CTO and co-founder of Pluto and member of the OpenXR working group. Pluto is a spatial communications company with the purpose to help humanity transcend physical location. Pluto has two clients, the PlutoVR alpha and an alpha for iOS devices with TrueDepth cameras.