Watch A Smartphone Turn Into A Controller For HoloLens

by Jamie Feltham • March 8th, 2017

Microsoft’s HoloLens is still very much in an early stage, and interesting new ideas and uses for the device are being discovered by the week. This most recent example turns your phone into a controller for the mixed reality headset.

Currently, HoloLens is mainly controlled through gestures with your hand, though can also be navigated with voice recognition and gaze-based input. To development agency AfterNow, though, that wasn’t enough. The studio wanted the kind of control you can get with VR controllers like the HTC Vive’s six degree of freedom (6DOF) wands but for HoloLens. Instead of using just a Vive controller, AfterNow turned to something millions of us already have, the smartphone.

The result is what you see in the video above. The user’s smartphone isn’t just a 6DOF controller but also a sort of virtual launching device. In the video, the user brings up virtual cubes on the smartphone screen, and is then able to flick them out to project them as objects in the real world.

It’s an intriguing potential means of control for the platform, but one that Microsoft itself might not agree with. As AfterNow’s Jesse Vander Does notes in a blog post, the company asks that “interactions should be a comfortable 1.5 meters from the user” to both reduce eye strain and also “ensure that the items the user is interacting with stay within the frame.”

“Nonetheless, there will most definitely be use cases that benefit from controllers,” Vander Does adds. “Moreover, there are a lot of fun things we could do with a 6DoF controller that also has a touch screen.” He gave examples including a sort of mixed reality Tilt Brush, where users might begin a line on the phone and then pull it through real space to make 3D images. You could also pull 3D assets into your phone.

While HoloLens is controller-free, Microsoft is also bringing VR headsets to market that can use physical controllers. We saw one last week that used an Xbox gamepad, but the company also notes that 6DOF controllers will be important to the platform.

Tagged with: ,

What's your reaction?
  • John Shull

    I’m familiar with developing on the Hololens and was looking into spending some time in this exact area. I recognize the Vuforia marker, are you also Bluetooth syncing the phone to pull back local gyroscope data and touch or did you network both of the devices via a local access point?

    • Jesse Vander Does

      Hi there, Jesse from AfterNow here. You’re spot on about the Vuforia marker. With regards to communication, the only thing that is sent from the phone to the HoloLens are the touch events. This is done over WiFi.

      I’ve thought about evolving this project by partially replacing Vuforia with a Tango enabled phone. I say partially, because I might use Vuforia to do an initial alignment into the HoloLens reference frame, then use Tango for extended tracking. This would free up the phone screen to actually display UI elements.

      • I’ve used Vuforia with HoloLens inside Unity and framerate was, ehm, terrible. What has been your experience?

        • Jesse Vander Does

          It seemed to be improved from their beta release, but yeah, not exactly smooth. ATM it seems to me like its better suited for non-realtime scenarios.

          • ultrafro

            Great work! Similar problems using input on hololens. I’ve tried streaming a vive controller, but I’ve run into interference issues when the controller is in front of the hololens (vive controller will stop tracking). Anybody figured out a way around this?