PianoVision: Practicing Piano On Quest With AR, MIDI And A Real Instrument

PianoVision presents the best version yet of an educational piano app for Quest. Not only does it overlay notes onto your real piano using passthrough and hand tracking, but it also responds to your playing in real time through connections with MIDI-enabled keyboards.

Available in Early Access, PianoVision feels like the natural culmination of many previous piano apps on Quest. As new features become available to developers, each successive piano app has added more functionality and made large improvements in one or more areas. PianoVision might be the best version yet.

PianoVision essentially lets you align a virtual piano to your real keyboard, and then uses passthrough to teach songs by displaying notes falling in sequence toward the keys.

This functionality by itself isn’t new — others have done it before –but the real killer feature of PianoVision is the ability to connect the headset to a MIDI keyboard or piano. Once connected, the app is able to tell which notes you’re hitting on the piano, responding to your playing in real time. If you miss a note or play the wrong one, the sequence stops advancing until you recognize your mistake and play it correctly.

This works through a connection with a desktop app for PC and Mac. The desktop app connects to your MIDI-enabled keyboard via USB, then communicates wirelessly with PianoVision on Quest in real time.

My housemate and I have both played piano for over 20 years, and both of us were impressed and shocked at how effective PianoVision was as a tool for learning and visualizing songs in a completely different format to reading sheet music.

I was able to run through Clair de Lune (a piece I’ve toyed around with in the past, but never learned properly) and was amazed at how useful PianoVision’s tools were to visualize sections that I hadn’t been able to crack with sheet music in the past. Being able to ‘see’ the notes falling onto the piano keys is a very different and tangible method of learning a song, and it works to great effect.

There’s a selection of classics, from easy to hard in difficulty, that come installed on the app, but you’re also able load in your own custom MIDI tracks through the desktop app (as long as they’re 2 track piano MIDI files). You can also adjust the speed, use a metronome and practice one hand at a time if you need to. Plus, each time you run through a song from start to finish, the app notes how long it took you — as you get better, you’ll improve on your high score time until you’re playing the song at tempo.

As impressive as the technology and functionality is, there are some disclaimers. For one, it helps if you already have some experience playing piano, even at a basic level. It would certainly be possible for a complete beginner to use the tool, but it’s not going to replace actual lessons anytime soon.

I expect most musicians would likely agree this may be best used as a supplementary tool rather than a compete replacement for traditional teaching methods. There’s nothing here that would cover other essentials of learning, such as music theory or proper technique. PianoVision helps with the notes themselves, but can’t help you with the why and how.

It’s also easier to use if you know the song in question. The length of the notes do indicate basic rhythm and timing, but it’s not ideal. I found it much more difficult to interpret the rhythm of songs I hadn’t heard before.

Even in Early Access, PianoVision is making huge strides for music education options on Quest. The next big missing link for this kind of technology is being able to provide tips on technique, both in a technical (finger and hand position, moving between notes fluidly, etc.) and expressive (dynamics, feel, emotion) sense. Given how complicated and subjective those are, that’s probably still years off becoming a reality. But for now, PianoVision is one of the most interesting VR-based music learning software we’ve seen as it presents a fascinating glimpse at the potential benefits and avenues of using VR as an educational tool.

You can sign up to PianoVision’s Early Access program here.