Creating realistic interactions with objects and people in virtual reality is one of the industry’s biggest challenges right now, but what about for augmented reality?
That’s an area that researchers from Massachusetts Institute of Technology’s (MIT’s) Computer Science and Artificial Intelligence Laboratory (CSAIL) have recently made strides in with what they call Interactive Dynamic Video (IDV). First designed with video in mind, PhD student Abe Davis has created a unique concept that could represent a way to not only interact with on-screen objects, but for those objects to also realistically react to the world around them.
If that sounds a little confusing, then we recommend taking a look at the video below.
At first, IDV looks like a pretty simple concept. Using a traditional camera to record the object in question, Davis observes the tiny vibrations that item will make, causing them to occur himself if necessary. That information is then recorded and applied, and the result is video in which you can grab and pull parts of the object, and have it move in realistic ways. The video above shows the user bending the arms of a wire man, for example, or swaying the leaves on a bush.
Where IDV gets really interesting, however, is when virtual objects are introduced into that video to realistically interact with the environment around them. In the video above, researchers create a simple dinosaur within a paint app, and then bring him into a pre-recorded video of a child’s jungle gym in motion. The dinosaur bounces from wire to holds, making his way down the structure. With every leap, the anticipated vibration on the jungle gym is realistically displayed.
Later on, we see Davis himself appearing on screen, holding out his hand in front of a bush and making a series of dramatic movements. In real life the bush obviously won’t move, but when IDV is applied to the video, he gets the bush to bend to his will. It’s very impressive stuff.
And, yes, you could theoretically apply this concept to Pokemon Go.
You’ve probably had a lot of sessions with Niantic’s AR phenomenon where a virtual beast as awkwardly stood on top of a bush without knowing it’s really there. With IDV, Davis suggests that Pokemon would be able to bounce from leaf to leaf of that bush, with the object moving as you’d expect. Davis called this concept Dynamic Augmented Reality (DAR). The problem, of course, is that you’d have to pre-record the environment you’re playing in and apply IDV. Maybe we could do that in personal play spaces, but to apply if to the entire world would be impossible right now.
Still, this is a fascinating concept, which could become consumer-grade if it were to be implemented into perhaps a smartphone app, or YouTube. This might be the beginning of a new means of control for AR applications.
