A Case For Pencil!: Why Physical Drawing Is Better In Mixed Reality

Pencil just released on Quest in Early Access. If you’re keen on the future of mixed reality for hobbies and education, and curious about pushing the boundaries of 2024 tech, give this one a try. 

Using a pencil in Pencil

Like it says on the tin, and like you’ll see in the trailer: in Pencil you draw with regular pencils.

Many drawing apps, including the former whiteboard in Meta’s Workrooms app, compromise: hold your controller in reverse and use its butt as your drawing utensil. It’s awkward. Pencil rejects the premise: Just use a pencil; you already know exactly how that “controller” works. The game doesn’t have to perform sophisticated calculations to discern the angle, direction, and pressure of your stroke; you do that all yourself. Mastering that "controller" is still up to you.

But you can already draw on paper with a pencil from your living room without a $500 headset, much less a $10-15 app. So why the recommendation?

As of 2024, it’s the killer combination of lessons and progressive outlines.

Leo, Pencil’s avatar, guides you in real-time through a portfolio of drawing techniques. But unlike the lessons you can find on Youtube, Pencil’s lessons place outlines directly on your canvas. Those outlines get more complicated as you progress, and you can toggle their opacity to check your own work.

Following an outline significantly improves my drawing — not just in the quality of my finished work, but in building my confidence and muscle memory about shapes & techniques. It's a tiny physical difference — sans-outline, I'd have to glance & reference my subject and rely on (very, very recent) memory to draw it, instead of drawing what’s directly in front of me. But after just a few lessons, I already feel far more confident.

Controls & controllers in Pencil

A decent MR innovation becoming a standard: In Pencil, place your left controller flat on your desk to convey your paper’s location to your headset.

Pencil is the latest to leverage this trick: Because Meta doesn’t allow 3rd party access to the front-facing cameras, apps will ask you to gauge the location of objects in your room by proxy of your controllers’ X/Y/Z-coordinates. (Meta’s own Horizon Workrooms does similar by asking you to dictate your desk’s height by controller resting position.)

Yet, I faced a challenge familiar to anyone who’s ever stenciled: If you nudge your paper or controller the *tiniest* bit, your outline may be nearly impossible to realign. It makes sense, come to think of it, that Pencil’s overlay depicts your paper being taped down.

I sort of lament that Pencil leaves the other controller unutilized. You use your off-hand and midair UI to toggle the settings & transparency of your outline, which is finicky. It’d be easier & more precise if you could use the buttons and joystick of the unused controller.

Import & Export

For creative experiences like Pencil, the Import & Export experience is crucial: Sure, it’s nice to draw in OpenBrush, but if it's a nuisance to get your creation elsewhere — into your desktop to further refine, or printed to display on the fridge, or sent to a friend — everyone else misses out.

Pencil has a seamless physical “export” since your work is already in the physical world. “Import” is another story: Getting templates from the internet into my game was a headache. And given the “tiny nudge” problem, I can’t imagine resuming & attempting to complete a WIP outline-based drawing. Non-outline drawing based on Pencil’s in-app 3D models would be no problem to resume from WIP, but I’m not convinced this is where the app’s real magic lies.

What should MR be?

Pencil’s success as an MR drawing aid points to fantastic potential ahead.

Access to front-facing cameras seems inevitable, and will unlock a superior experience in Pencil. Accurate tracking would let me resume WIP works. Better: Perhaps it’d empower Pencil to critique my work. I’m reminded of the old Mario Party minigame “Crazy Cutter,” but my actual drawing skills are graded instead of my ability to steer a joystick. Leo shouldn’t just give me a final score, but also a live score with suggestions on what to fix — my pencil *does* have an eraser, after all.

This Pencil concept has room to expand beyond its position as a drawing aid. Handwriting springs to mind and perhaps even arithmetic practice would be more effective or fun with a character like Leo guiding. Maybe there’s room for surprising mixed reality games: What if a Pencil-version of Fruit Ninja had you slash digital objects by swiping your pencil along your paper, and something artistic came out as a result?

We’re headed for a future where physical objects — even “dumb” ones, like a regular No. 2 pencil — can be brought into digital play, and where mixed reality can elevate (not just digitize) familiar activities. Pencil is an early look at what that future might be like. So for VR and MR enthusiasts: Pencil will stretch your imagination, and you may even pick up a new skill or hobby along the way.