Oculus loves collecting computer vision companies almost as much as Palmer Luckey likes collecting HMDs. Today Oculus announced they have acquired Pebbles Interfaces, a computer vision company out of Israel. As of now, Oculus is not disclosing information as to the deal’s terms, but the Hebrew language site that broke the news, Calcalist, reported at the time the deal was worth at least $60 million (although the post has since been removed) which would seem about right given the $11M series B raise the company made in August 2013.
In Oculus’ official blog post on the subject, we are told very little about Pebbles beyond that they “[have] spent the past five years developing technology that uses custom optics, sensor systems and algorithms to detect and track hand movement,” but it would appear that the acquisition may mean more than just more researchers in Michael Abrash’s lab in Washington. “Pebbles Interfaces will be joining the hardware engineering and computer vision teams” at Oculus, which means that some of the team will focus on the software side while the rest of the team is focused on developing camera technology for tracking and object detection at Oculus. While this is not stated outwardly, it makes sense given Oculus’ direction.
In the video above we get a brief glimpse at the camera that Pebbles Interfaces are using to track the hands, which would appear to be the “custom optics” mentioned in the post. The camera appears to be something that developed at least in part by Pebbles themselves, and if their YouTube video from 2012 is to be believed was the first ever depth gesture camera created. According to an archived version of Pebbles’ website, “[they] develop advanced motion sensors that display physical objects within the digital space, at any range or angle, with no latency.” Notice in that that statement didn’t say hands but rather objects, this is very important when considering the broader strategy.
Take a look at this gif (above). Notice anything interesting? That card in his hand has no fiduciary markers and yet is being tracked in space, and not through an AR ghost passthrough either. This is a very interesting feat, which is also demonstrated with a cube later in the video (showing the scanning can work with full three dimensional objects as well). What this means is that Oculus has just acquired the company that could help enable not only hand tracking (which is prominently on display in the video) but also could eventually lead to a full object detection system that would be necessary for full room scale experiences that don’t require you to move all the furniture out of the center of the room.
But the star of the show, at least for now, is the hand/finger tracking which appears to be very robust. In the video we saw a number of demonstrations of very complex actions that simply aren’t possible with any of the tracking solutions on the market currently. Of particular note is the software’s ability to apparently seamlessly handle incidents where occlusion occurs. Typically this is one of the things that can break tracking faster than a bull breaks plates in a china shop but in the gif below you can see that Pebbles’ software holds true:
As you can see the hands are not only able to pass over one another, but also interlock all with minimal added artifacts (although there are definitely still visible artifacts). It is worth noting that the images here are of the depth mapping that is being applied to the hands, rather than a model that is linked to the hands motion, which can be much more difficult to pull off with this kind of tracking stress (the artifacts would likely break the tracking on the model still). But this is still very interesting.
One last thing to add is that it may be more than just hands, fingers and objects that are trackable with this solution. In this interview with Geektime, Renana Perlovitch said “Pebbles is much more than just gesture, we’re a human sensing company… we have a very unique technology that actually enables you to have your own body parts inside the VR experience.” Yup that’s right, full body tracking – at least according to the company’s representative. Near the end of the video you can see some of this demonstrated quickly, with facial tracking:
It is clear to me that Oculus’ acquisition of Pebbles Interfaces is to serve four purposes: to help support the teams focusing on inside-out tracking solutions using a forward facing camera on the Rift/GearVR, to bring object detection into that space so that you can safely move freely through physical space while in the virtual world, to work on finger and hand tracking with the Nimble team that Oculus acquired back in 2014, and finally to eventually bring one’s full presence into an experience.
This acquisition only further solidifies Oculus’ commitment to computer vision as the tracking solution. Armed now with four companies that are focused on computer vision (Surreal Vision, Nimble Labs, The 13th Lab and no Pebbles Interfaces) and a partnership with Microsoft – who are doing some epic things in computer vision themselves – it would seem that if anyone is going to perfect a solution for hand and finger tracking as well as object detection, it is going to be Oculus.
