Jaymis Loveday’s most recent mixed reality video project was filmed with a helpful robot camera assistant.
Loveday seems to be something of a mad scientist working in Australia, creating the first music video built in Tilt Brush. Recently he got access to a third HTC Vive controller, allowing him to start experimenting with mixed reality video capture. With Loveday’s most recent experiment, he essentially built a camera robot assistant.
Loveday filmed artist SUTU as he sketched out a scene in Tilt Brush over the course of about four hours. The scene is from a panel SUTU previously made for a 2D interactive Web-based comic book called NAWLZ. In the video below, you can see him place a reference image in the scene as he begins. SUTU was previously commissioned by Google to create a work of art in Tilt Brush, so he knows his way around the sketching program.
The video represents a series of experimental camera techniques. The result is a kind of cinematic stabilized camera system that can be operated with no human involved, or handheld by one person who is seamlessly capturing the action in two realities. Powering the system is Valve’s SteamVR Tracking technology and a DJI Ronin gimbal receiving commands from an Arduino telling it to keep the VR headset in the frame.
“I’ve written some software in Unity which takes the position/rotation of the mixed reality controller, and figures out pan/tilt angles to the headset,” Loveday told UploadVR. “It’s running as an “overlay” application, so it can work alongside a regular piece of VR software. Unity sends those angles via serial to the Arduino, which sends pan/tilt commands via RC transmitter to the Ronin. So for the timelapse stuff, I make it so the movements are slow, and there’s a bigger “dead zone” so he can move around and it’s not jerking everywhere. And for the real-time stuff where I pick up the gimbal, it’s moving faster so it keeps the headset centered.”
The experiments were captured without a green screen, an addition which could make it easier to add the virtual environment to the scene. Still, the video — particularly shots near the end showing a mostly opaque work of art in the foreground — show enormous promise on the path to impressive mixed reality videos and broadcasts. Next, Loveday will be looking at how to make the system work with cheaper, smaller hardware.
“One of my next steps is to have the robot physically moving around the space as well,” Loveday said. “Initially it’ll be on a 6 ft camera slider, but there are other options for robotic moving cameras.”
