Hands-On With Valve’s Knuckles Prototype Controllers

by Denny Unger • July 3rd, 2017

Editor’s Note: Valve has started shipping the prototype Knuckles controllers to select VR developers in limited quantities. UploadVR hasn’t gotten its hands on a pair just yet, so we’ve reached out to the development community to see what they think. This article is a contribution from Denny Unger, Co-Founder, CEO, and Creative Director of Cloudhead Games, the studio behind The Gallery: Episode 1 – The  Call of the Starseed and Episode 2 – Heart of the Emberstone


User interfaces in virtual reality start with your hands. We know that now with the Oculus Touch and the HTC Vive, but even when VR was simply a screen strapped to your head many felt that hands were the future. We began to develop The Gallery on the first Oculus devkit with the Razer Hydra (a Sixense technology) to deliver surrogate hand tracking and body presence back in 2013. And when Valve invited Cloudhead and the first wave of developers to see what would eventually be known as the Vive, Valve showed a commitment to that same vision. Now Valve has invited us and a new wave of developers to begin working with their latest prototype—the SteamVR Knuckles, a wearable VR controller that tracks not just your hand, but each finger too.

Back in 2014, we didn’t realize exactly how accurate SteamVR tracking was—the whole notion of roomscale VR was almost incomprehensible. Up until that point, there was still some motion latency in VR, so you never felt completely attached to the actions in your hands. But once the SteamVR Lighthouses started tracking objects in a 3D space, it was a deep and immediate connection of, “Holy crap, that’s actually my hand in VR.”

Our goal then was to deliver an experience where the player doesn’t have to think about the controller, and has only natural, gestural interactions. We wanted to demonstrate why this kind of input—your hands themselves—was meaningful. When we received our first Vive devkit (wired at the time) we were taping them to our hands in order to feel more immersed, and we even spoke to Valve about crude ways they could strap the controller on.

Those early Vive prototypes already showed an incredible level of fidelity, capable of measuring the tiniest fraction of a movement. It’s like the Moore’s Law of motion control; each incremental improvement in tracking brings with it new possibilities. So as soon as you have that kind of fidelity with your hands in VR, you need your fingers to be more purely represented. And that’s what the Oculus Touch started to do by bringing capacitive, gestural input to the controller.

Where Touch differs from what the Knuckles offer, however, is that you’re still pushing a binary button in the end; Touch feels more grounded in traditional gamepad design. Specifically, you always feel like you’re holding something with buttons—and that works perfectly for gun games and sword games. But the Knuckles take that further by removing the abstractions of first-gen VR inputs. Even though it looks like a more complicated device, it’s actually a much simpler one.

With the Knuckles, you’re not holding a controller; it simply straps to your hand and rests in your palm. If you relax your hand into a natural flex, the controller stays put and keeps tracking your fingers. If you reach out to grab an object in VR, your hand wraps around the base of the controller, giving a tangible feeling of grabbing something. That physicality is something you don’t get from data gloves, or vision based inputs without any device, and that feeling can then be fine-tuned with haptic feedback. Plus, you’re not passing around a sweaty data glove between your friends.

When reaching out to an object with the Knuckles, I’m not thinking about the controller I’m holding in my hand, because I’m not holding one. I’m not thinking about how to use my fingers either, because they’re not assigned to a button press. I’m not even thinking about my hand, and that’s where the magic comes in—I’m just thinking about grabbing the object, as I would in real life. That entire grasping motion is represented in VR, whether I pinch with two fingers, scoop with my hand, or close my fist around it. The Knuckles track your fingers by the distance they are from the base of the controller (your palm), and represent that movement in VR. It’s second nature.

As developers, when we receive prototype hardware like the SteamVR Knuckles, it makes us want to push the capabilities. In the past, that’s meant radically rethinking our stack of interactions and locomotion systems—virtually redesigning the game. We’ve written about what these kinds of changes have meant for The Gallery in the past, but the long story short is that new controllers like the Knuckles aren’t just affecting the complexity of interactions. We now have new possibilities for game mechanics and design that haven’t and couldn’t have been done before. It’s to be seen how these controllers will impact Heart of the Emberstone in September, but they’ll be a core focus in designing Episode 3.

Think of an interface-heavy app like Tilt Brush. Dials can be intuitive, but using your fingers is organic. There’s a possibility for gestural movements to call functions and navigate dense data; there could be an entire language built out of using your hands to manipulate paint brushes and pencils and sizes and colors. Once you take the mental load of an interface off the player—once they stop thinking about the controller—you can leverage that partition into experiential design and organic controls.

The kind of technology that the SteamVR Knuckles offer is not just impactful to the future of input in gaming, but also the future of output. I can go to a social VR space and point to something, or offer a peace sign, or tell somebody to hang loose without having to think about it. The controller doesn’t guess your gesture, or snap to a new position, it represents your fingers based on the distance it calculates. The more natural and intuitive the interface, the less we think about hardware. And the less we fixate on hardware, the more present we can be in VR.

Prior to the Knuckles, hardware developers were looking for something that would be more broadly accepted by the general public. Something that resembled a Wiimote, like the Vive wands, or something that when put together resembled a gamepad, like the Touch controllers, meant that VR input was familiar. Strapping an alien device to your hand in first-gen VR would have been too much too fast. But I think it took the evolution of those two controllers to get VR to the point where the public could be comfortable with the idea of a controller strapped to your hand.

There are so many moments in life in which using your hands is a vital part of the experience. There are implications for education and communication—with audiences who don’t generally understand videogame controllers—because the SteamVR Knuckles open the door to that broader audience. These are pick-up-and-play controllers where you don’t have to think about the input, you just reach out and interact in virtual space.

User interface in virtual reality starts with your hands. And once users are empowered in that way, and don’t have to be told how to use the technology, the next generation of virtual reality is here.

This is a guest post not produced by the UploadVR staff. It’s a contribution submitted by Denny Unger, the Co-Founder, CEO, and Creative Director of Cloudhead Games. No compensation was exchanged for the creation of this content.

Tagged with: , ,

What's your reaction?
Like
Wow
0%
LOL
0%
Dislike
0%
  • NooYawker

    Finally I can give someone the finger in VR. But I don’t know how secure it really is taking all your fingers off the controller. What if threw a virtual baseball pitch will it really stay on?

    • Reuben Welsh

      If you took all your fingers of a baseball bat, would it stay firm?

      This is going to react pretty much like anyhing real, of course you will have to grip and wont be able to use gestures while doing over rapid movements, Try giving someone the finger while swinging a baseball bat, im sure the result will be similar to real life 🙂

      • Caven

        I think you’re misunderstanding NooYawker’s question. We’re not talking about a baseball bat, we’re talking about a baseball. I’m I’m using a Knuckles controller and I’m holding a baseball in VR, I could conceivably throw that baseball by performing a throwing motion, and then releasing my grip on the controller just as if I were releasing my grip on the virtual baseball. However, a crucial difference is that I want to have the virtual baseball leave my fingers at high velocity, but in the real world I DON’T want the Knuckles controller to leave my fingers at high velocity. Does the retention mechanism on the Knuckles controller work well enough that it will remain attached to the hand–even if a person has released their grip on the controller in order to throw a virtual object?

        • Reuben Welsh

          ah, see now that i missunderstood. Maybe they should add some kind of strap. It will be hard to have a comfortable controller while alowing that kind force from throwing a ball. Only option i see would be a glove or something that you place your hand in. But gloves get pretty nasty.

          • Duane Locsin

            Maybe if they added an additional strap that went around your wrist to make it more secure could do it.

    • mikowilson

      I’ve pretended to do just that, and they stay on firm. It all depends on how aggressively you tighten the cloth strap.

      • NooYawker

        That’s what I wanted to hear.

  • Doctor Bambi

    It’s great to hear his perspective as a designer and the impact these have on interaction with VR worlds, but I hoped he would have discussed more about using the technicalities of using the hardware itself. I’m sure that will come in due time, but darn it, these controllers are super cool and I’m hungry for all the details.

    How easy is putting on the second controller?
    How easy is putting on a headset with the controllers on?
    How about typing on a keyboard, using a mouse, or picking up a drink?
    How reliable is the tightening system, does it feel solid? Any slippage over the duration of a play session?
    Any issues with discomfort after using them for while with pressure along the back of the hand?
    How about tracking? Do they hold up to occlusion as well as the wands?
    Did people of differing hand sizes give them a spin and were they comfortable/usable for most hand sizes?
    Any issues with heat build up underneath the straps? I’ve noticed sometimes my hands get sweaty with the Touch controllers.

    These are the things I’m most interested to learn about at the moment. I believe there’s nothing here too glaring, but it would be nice to get some confirmation on some of these points.

    • Antony Peraza Stevens

      1. Pretty easy.
      2. If you’re used to holding controllers in your hand while putting on a headset (say Oculus Touch) not much harder than that.
      3. Typing is possible, but you have to poke. Mouse and picking up a drink aren’t really things because you don’t have a palm.
      4. Tightening is reliable, but I wouldn’t hurl a baseball. Lobbing feels very safe.
      5. No. They’re most snug on the palm.
      6. Tracking is about the same as far as I can tell.
      7. Yes. Big hands had the most trouble getting comfortable and getting a good reading with calibrations. Small hands struggle to reach the top of the trackpad.
      8. I get sweaty very quickly with Touch. Haven’t paid attention to sweat with the Knuckles, but they haven’t started slipping from sweat or anything.

    • Denny Unger

      How easy is putting on the second controller?

      Super easy because once your first hand is cinched, the controller is locked to it freeing up your fingers to do whatever.

      How easy is putting on a headset with the controllers on?

      Again pretty easy. Whereas before you’d have to put Vive controllers on the ground or table, now you can attach controllers first if you want to and pull on the headset.

      How about typing on a keyboard, using a mouse, or picking up a drink?

      Using the keyboard; yes but only pecking with a couple fingers. Mouse, the same. I wouldnt pick up a drink with much confidence here.

      How reliable is the tightening system, does it feel solid? Any slippage over the duration of a play session?

      Thus far super reliable, feels quality, keeps its grip even when doing overt chopping actions.They did a great job with that whole mechanism.

      Any issues with discomfort after using them for while with pressure along the back of the hand?

      Nothing overt that I’ve noticed so far.

      How about tracking? Do they hold up to occlusion as well as the wands?

      Easily yeah. Again, they nailed sensor placement.

      Did people of differing hand sizes give them a spin and were they comfortable/usable for most hand sizes?

      That’s a bit up in the air. We have one person at the studio with shorter thumbs so the thumbpad is not entirely within comfortable reach for him.

      Any issues with heat build up underneath the straps? I’ve noticed sometimes my hands get sweaty with the Touch controllers.

      There might be but I haven’t noticed excessive moisture either. I need more time with them to have a better answer there.

      • Doctor Bambi

        You’re the bomb! Thank you for answering!

      • 13penguins

        I have brachydactyly type D, a.k.a toe thumbs and I’m hoping I’ll be able to comfortably reach the thumbpad.

  • Sean Lumly

    Nice writeup! Top stuff.

  • impurekind

    Controller design looks rough at this stage, but the overall functionality is looking very cool.

    • M0rdresh

      Yeah it does look like a wand 2.0 to me, I expected something less bulky.

  • Adam Hubbard

    I don’t want to say they’re copying Oculus Touch controllers, but let’s just say that Oculus did it right the first time. That being said, I’m a fan of both headsets, I just wish people would realize that Oculus is formidable competition.

    • Protowalker

      This has fully analog finger tracking, though, and allows you to use your hands to interact in VR. Unless every single dev is lying to us, this is better than Touch.

      • Duane Locsin

        Oh well, the good thing that comes out of this is that competition gives incentives to do better designs, refinement and push the technology even further.

        I would be more concerned if there was no competition and we would still be using game pads.
        Even the mobile browser space it started with controls on the headset, then a controller, now they have a Vive like wand(s)

        Not to undermine the amazing direction of hand controls, I wish there was as much emphasis on natural locomotion as well.
        I have only seen three major attempts for treadmills – Cyberith, Virtuix and Infinideck but they look to be longer process before they are ready to go prime time.

        Virtual Reality – I guess we have to learn to grasp, then crawl before we can walk.

        • M0rdresh

          Treadmills are a danger to the VR industry – this has the perception of making it all gimmicky and niche. They should invest in better artificial locomotion, I don’t want to actually walk 8km in Fallout 5 VR in 2024 to get to the quest npc.

          • Duane Locsin

            Well you don’t need to walk 8km in Fallout 5 VR there are other locomotion options available.
            For me personally I am all for the additional exercise as it adds to the immersion factor.

            Walking, and kneeling behind a table hiding from bandits in an abandoned building manually reloading your gun is surreal.

          • That sounds better in theory than it is in practice. I’m not severely out of shape or anything (5’11” @164lbs; medium-elf-like-build), but after about an hour of playing Farpoint while standing… man… I was getting sore. But that might be partly just me… I’ve been in a couple of car wrecks over the last decade that left me with nerve damage.

            But yeah… I’ve been thinking about this quite a bit since then; what the future will hold for immersive locomotion that feels as real as we can get –reasonably, and I’m beginning to think that physical devices that rely on human locomotion rather than controller interface-based locomotion is going to be much harder on people to use for extended periods of time, especially after working all day –when I get home in the evening, I just don’t have the energy to walk a couple of miles. I’m hoping that we can find some sort of middle ground that works great for everyone, but I’m also in support of opening things up to allow people to use whatever they have available –meaning that if a physical-locomotion device is detected then it can be used, but wouldn’t be a requirement by any means –instead opting for a universal API to allow interfacing of multiple types of locomotion input.

    • polysix

      it would be no different to oculus copying valve’s cutlass prototypes (way before either released), oculus had used valve’s controllers which were very simliar to what the touch became (in shape and function minus finger stuff), but valve abandoned that shape cos occlusion sucked. Valve got it right, look at the issues facebook has had, and how long it took them, to get tracking even close to how good valve’s is. Facebook has copied valve in many many more ways than the other way around, from dual screens, low persistence … all this was on valves early prototypes and oculus used those (thanks to a generous valve) and ‘copied’ much of it to their own HMD. Frankly am sick of hearing from VR noobs who only arrived post CV about oculus being ‘copied’ when they don’t know the back story.

    • NooYawker

      People may talk smack about oculus because it’s owned by Facebook but of course it’s formidable competition. It’s a great piece of hardware and yea, thought I never used the touch controllers I imagine they work great and they’re a much better design than vive wands. But they are owned by FB so screw em. 😀

  • wheeler

    Looking at some of the videos, it seems like there’s occasionally a delay between physical and virtual finger movement. I’m guessing this is because the controller needs to guess where your fingers are based on points on contact on the capacitive sensors so it isn’t always exact. On top of that it seems like there is some smoothing/interpolation between movements which could be another source of delay.

    What I’m wondering is if better calibration techniques could yield something more 1-to-1. For instance, imagine an interactive calibration program that instructs the user to perform a variety of gestures and confirms the results at each step, and then derives everything else from that and saves that data in a profile. Of course a quick calibration method is essential, but something with higher fidelity also seems like it would be desirable .

    I’m also wondering what sort of interaction motifs devs will settle on. For example, with grabbing something: is it better to consider an object in an attached (grabbed) state when the tracked fingers are simply clasped around its virtual representation? Or should the user have to fully clasp the controller? I’m thinking the latter since the lack of physical feedback would result in ambiguities that would only lead to frustration. But then how does one deal with the disconnect that results from one’s fingers passing through the object? And for experiences where one is almost always wielding an object (e.g. gun or sword), I’d imagine that devs would need to carefully define grabbing/dropped criteria or else users will find themselves accidentally grabbing and dropping things quite often.

    Sorry if this sounds like a critique of the controllers–in actuality I’m very excited about them but my mind goes straight to the implementation details and the challenges of making the best use of a controller that is half way between a gamepad and leap motion. Depending on the situation, devs are going to have to decide whether or not they should constrain inputs for highly predictable results or if they can forgive some inaccuracies to allow for something more dynamic. But it’s great that they now have both options. Plenty of opportunities and challenges with this thing.

    • Martin L

      The delay is mostly due to the video footage and the in-game footage not being in sync. This has been confirmed by Cloudhead Studios. Similar “problems” and latency were pointed out initially when the motion controllers were first demoed, but again, that was because the footage isn’t in sync.

  • J.C.

    Excited for these to come out! I do worry that it’ll make interacting confusing when going back to older games that devs don’t patch, but that’s the price of progress.

    Also, my concern about handing over sweaty VR gloves is apparently not mine alone. VR gloves may be OK when everyone has their own setup. Right now, I already intentionally avoid inviting people to try out my Vive if they tend to sweat a lot. But hand sweat can happen as a response to fear, so it’s hard to judge if someone would nasty up a pair of gloves.

  • Tomas Sandven

    It’s times like these I wish I could just go to bed and wake up in 6 months

    • J.M.Wagner

      lol!

  • bereanboarder

    I wonder what a person who is fluent in sign language would make of this technology? Would they be able to help with some common gestures that could translate into operating system actions? I have no answer to this question, I just wonder if we should reinvent a “gesture wheel” as it were, that is perhaps already spinning and been refined?

    • Duane Locsin

      I can already conceive of some uses of refined hand/finger tracking.

      -Military gestures to your squad
      -‘Minority Report’ style interface commands for virtual interfaces
      -learning sign language (you input the hand gestures and it says what you are communicating)

  • Nice article, but it’s not a “hands-on” session. It doesn’t describe all characteristics of the controllers. We all could already imagine all this stuff: what we want are videos, photos, experiments, SDK description, etc…

  • Duane Locsin

    It has to be said.
    With this kind of granular gesturing and hand tracking there will be at least two major used gestures initially.

    The peace sign and the other “not so much peace” sign😝

    It’s like learning a new language and the more “colorful” words are sometimes used first.