Watch The Microsoft HoloLens Project a 360-Video Into the Real World

by Alice Bonasio • August 17th, 2016

Admittedly, at first glance the HoloLens video featured above might not look like mind-blowing stuff, but this isn’t about the quality of the visuals. Instead, this is about where the images are coming from, how they were generated, and where they’re being projected.

Jim Reichert, a Senior Creative Technologist at Microsoft, flagged this up to me as he’s a huge fan of Second Life, an online, multiplayer social virtual worldand was excited when last week the first 360 video footage was recorded in the SL Metaverse, featuring Jo Yardley’s authentic rebuild of 1920s Berlin.

Since Jim’s role at Microsoft involves overseeing ‘next gen experiences’ (one of the coolest job titles I have ever heard of), he wanted to check out how this footage would look when projected onto HoloLens. As this was a quick pilot, he decided to lower the resolution and frame rate so he could iterate it faster, hence the rough-and-ready look.

SL Video 1

The result you see in the test at the top of this article is therefore 360-degree footage, which had been recorded live in a virtual environment (Second Life) and then projected back onto the augmented reality (AR) gear, superimposing it, in real time, with the real-world background of Jim’s office. And if that doesn’t make your head hurt just a little bit, you’re not doing it right. The possibilities it raises are tantalizing.

This is effectively a proof-of-concept for how easy it will become to make user-generated content for both VR, AR and Mixed Reality in the future.  The SL film shows a train pulling into the 1920s Berlin station, which is a direct reference to the first-ever movie – Arrival of a Train at La Ciotat – by the Lumière Brothers.

Berlin #2

The in-world footage was shot by Draxtor Despres, Phil Toma and Arduenn Schwartzmann using a simulated goPro Rig of 6 cameras, and they say that the workflow of filming, exporting, stitching and correcting that 360 footage from Second Life was actually faster than doing so in real life.

That opens up all sorts of possibilities for users to create their own content, as they would be able to use this technique to insert virtual sequences into live-action 360 video without having to painstakingly construct it element-by-element with CGI. The advantage of Second Life is that all the assets are already in place, so you can effectively film in real time. There is still a need to use a render engine, but that is no different than filming 360 video in the real world, which evens out the playing field.

Second Life creators, Linden Lab, are currently busy developing their dedicated Virtual Reality platform Project Sansar, which is set to launch to the public next January, and will also focus heavily on user-generated content. Yet this goes to show that, far from being obsolete, SL still has a few neat tricks up its sleeve. With a track record of over 13 years of creating virtual experiences, the Metaverse is the granddaddy of Virtual Reality, and it still provides an important creative outlet to an engaged community of more than a million users.

SL Video 2

“I see a lot of wonderful learning coming out of the Second Life residents, not just Linden Lab,” says Reichert.  “Even today, there’s no better place for artists and scientists working at the fringe of what is possible. It’s time for people to give Second Life a second look.”

Alice Bonasio is a freelance writer with work appearing in well-known publications such as The Huffington Post, Newsweek, Playboy, and more. You can follow her on Twitter: @alicebonasio.

Tagged with: , , , , , , ,

What's your reaction?
  • Wolf Baginski

    I am not sure what Project Sansar will end up giving us that Second Life doesn’t. Maybe it will just be that the starting point will be at a higher level, there is still a lot of obsolescent material in Second Life, and on what I experience, the more recent developments, the new ways of making content in Second Life, have not always been that well explained, and not always been well used.

    The people responsible for this video are the ones who know what they’re doing.

    • Phil Thoma

      Thank you. The original and first 360 video from within Second Life can be found here:

    • draxtor

      Wolf I hope down the line we can share a workflow and the tools involved so everyone in SL can make those videos. Loads of obstacles to overcome still though….

      • Wolf Baginski

        It’s apparent from the video commentary (six simultaneous cameras) that this is not going to be easy to arrange. And maybe some of the popular viewers for SL are wrong for this, there are computer resources they don’t need to use (I have two viewers that have a 2:1 ratio in memory use, but the extra features are useful in the fattie). Add the internet connection. I suspect my dependence on a few thousand miles of trans-Atlantic wet string is going to be a barrier, and keeping in sync with somebody in the USA would be a problem.

        They’re maybe talking about some local things, but a lot of VR proponents seem to expect a 40ms ping time. It makes some sense for some reactions to head movement, and I have seen reports of the Oculus Rift pulling a few tricks to offset the image axis within the headset. But the speed of light isn’t just a good idea, it’s the law.

        We can still make videos like this, it’s essentially passive, but I live within 150 miles of London and that’s 40ms away over the internet. It it going to be possible to get two-way VR interaction where people are doing more than sitting around the same table?

  • AcroYogi

    This is ridiculous to show in HoloLens. The entire point of AR / MR is to *fuse* virtual realities / holograms with physical realities / architecture. *IF* this person was in the Berlin train station modelled, accurately geo-tracked, and matched to real world geometry, that would be far more compelling… however, it appears that they’re in an office, with no physical relation to the projected movie. This *might* be a compelling experience in pure VR (at 90fps). In AR, as shown, it makes no sense at all.

    • Alice Bonasio

      I take your point, but as I said in the article, this was a quick prototype to test how you could superimpose that type of video in that type of hardware. Next steps are of course doing a lot of work to make the content actually make sense as an experience, that will come, I’m sure. In the meantime it’s interesting as a proof of concept, also to get people thinking of how it could be developed.

      • John A. Rupkalvis

        Very good illustrations and commentary (most stuff about the Hololens in the past has been misleading and confusing). However, no matter how good the description and graphics, you cannot really completely understand the Hololens until you have actually worn one. Being AR (with a nod to VR), it is also about perception, timing, and personal interaction. I am personally very interested in the mixed reality possibilities. I am working on concepts that involve being able to make real time transitions between VR and AR in the same headset, even during a single experience (dissolving between the virtual world and the real world, depending on either the virtual content, or real world activity, or both). I would like to communicate with anyone else interested in this aspect, especially from the standpoint of writing and coding gesture-sensing that would include dissolving during other stereoscopic interaction. You may reach me at [email protected] .

    • James T. Reichert

      It may not leverage the value-add of mixed-reality in the HoloLens at this stage, but I don’t think it is unreasonable to enjoy 360 videos in a HoloLens– especially if they’re just a backdrop to a more immersive, grounded-with-SLAM, near-field.

      I think that the “HoloTours” application amply demonstrates the value of having an untethered 360 video experience in a HoloLens, as evidenced by the reactions of anyone who has been transported to Rome or Machu Pichu. As you might gather from that application, volumetric holograms can be placed and oriented in the space– and interacted with. The 360 video is simply a way of creating a far-field envelope, much as you might do with a skybox.

  • Ran Hinrichs

    If anyone knew the graphics computing power required behind the seemingly effortless connection between the physical world and the virtual world, it would blow their mind. Jim is right, this is early days, but man are they progressing very fast.