For the creation of the Puzzling Places VR game on Oculus Quest, photogrammetry expert Azad Balabanian became a puzzlemaker while Shahriar Shahrabi worked as a programmer, game designer, and art director.
Puzzling Places is essentially the successor to a 2016-era virtual tourism app called Realities on Steam that was pitched with the line “Explore places that were out of reach before.” Puzzling Places actually started as an accident — a 3D mesh imported into disarrayed pieces on the computer like a jigsaw puzzle. Building from the initial hook for the game, the developers took a prototype to SideQuest, built up a supportive community on Patreon that was ravenous for more puzzles, and finally moved into beta testing release as a debut project on Facebook’s App Lab distribution system. This week, the photogrammetry-based puzzler became a fully released consumer software product priced at $14.99 with plans to hit Sony’s PlayStation VR platform before year’s end. There are 16 puzzles included at launch with difficulties in pieces including 25, 50, 100, 200,400, each with audio soundscapes that fill in as you solve the puzzle.
Shahrabi and Balabanian — two members of the six-person Realities.io team who built Puzzling Places — joined our virtual studio this week to trace the path of the game from idea to realization. We’ve got a full transcript of the discussion below which is broken up into chapters marking the key questions we covered so you can just jump to the part which interests you.
Can you explain how Puzzling Places was developed?
Ian: We’ve got a live interview today with the developers behind Puzzling Places out now on Oculus Quest. Tell us about the app and your path to release. You’ve got a very different story there than most VR devs. Can you explain?
Az: Yeah, I’m Az Balabanian and I’m joined with my friend Shahriar Shahrabi, we’re two developers out of six on the team and yeah, this game is quite a journey. We started as a prototype on Sidequest about a year and a half ago, it was a single puzzle that we released. We were very happy that it was very well received. Eventually we also created a beta for App Lab and it was one of the launch titles. And the entire time we were trying to get on the Oculus store, but our pitch was, let’s say, not being accepted. And eventually after enough people left amazing reviews and talked about how much they love the game and how cool of an idea it was, we were able to launch the full game on the Oculus store.
Is Puzzling Places coming to PSVR?
Ian: Also confirmed for PlayStation as well?
Az: Correct. And Q4 of this year we will have a PlayStation VR version of the game coming out.
Is Puzzling Places coming to Steam or PC VR?
Ian: What about PC?
Shah: We are thinking about PC, we would love to have it on PC. But it’s all a matter of resources and depending on what’s going to happen, our priorities are at some point you would have it on PC, but nothing is decided yet when and where.
How did Puzzling Places start as an accident?
Ian: Tell me that original story about this kind of starting off as an accident, is that correct? Can you explain exactly what happened there?
Shah: Yeah. So this, has something to do with the way we divide topologies and environments in order to have them run performant on a computer. And I was importing them in Unity back in the days, and something went wrong with the import and the pieces were flying everywhere. And I made a joke that we should make a game about this. So people come and fix it for us so that we can continue in development. And it just took a few seconds where everyone was just silent and they thought, ‘but that sounds like a really good idea.’ You know, we have a technology for it. We have all the measures and, yeah then that’s where the first prototype came from. Long time later, we actually just decided to really focus our development effort.
Az: That was back also on, I guess that was a PC sort of prototype and to have something run on Quest took quite a long time. And, so that’s where we are today.
How many puzzles are in the game at launch?
Ian: How many puzzles are in the game and what’s the update plan?
Az: The base game has 16 puzzles all with five difficulty variants, 25 pieces, 50 pieces, a hundred pieces, 200, 400. So it’s very much an accessible game to all in terms of the amount of hours you want to put into a single puzzle. 25 is a great place to start, but we have quite a lot of players that want even more pieces other than 400. You’ve got to have entire range of difficulty variants in terms of content and updates we’ve been developing this game for a year and a half. It’s had quite a lot of updates essentially since the prototype originally. And this is all due to our community helping us and giving us feedback about how they like the puzzle. We have quite a few ideas for the future. None that we’re ready to announce yet, but the way we’re thinking about this as this is the launch of a platform and anyone that has the game will get free feature updates, we will also have free and paid content that will come out. On Patreon over the last year and a half we made about 60 puzzles, one a week even. We want to make more puzzles. Absolutely. It’s just a matter of when.
Shah: Basically what we agreed on is as far as people are willing to play, people want to play, we will be releasing new puzzles.
Can you explain how the puzzles are made?
Ian: I’m seeing people asking the questions about the way puzzles get made and wanting to understand, is this going to be an automated process in the future that you’re going to have whole puzzles made from whole cities that you can easily put together a Google earth style, or is there a lot of hand human work done in making sure that these puzzles actually come apart and go together in a logical way.
Az: Currently it’s sort of a mix of both. We certainly have a puzzle making engine, which is a pipeline that we developed over the last few years, it wasn’t developed to create puzzles actually, it was developed to make photogrammetry run on mobile devices. And as a result as a by-product, the puzzling came out of it, as Shah talked about pieces being cut and jumbled. For now it’s absolutely hand curated puzzles. These are either scans that we’ve made, or scans that our partners have made. On the automated side, to create the puzzles, that’s an engine that we have that we put it into, but there’s quite a lot of work that goes into like handcrafting a puzzle to feel good for it to look good, for it to have the right pieces and get rid of bad pieces. So that for now is absolutely something we want to have control over, but in the future, let’s just say photogrammetry and scanning is starting to become a bigger and easier thing, more accessible. And would we like to support user generated puzzles, perhaps?
Ian: Do you think you’re going to be able to do a capture with a phone and break that apart in an automated way to have a puzzle? You’ve done some of these captures with drones, right?
Az: Some with drones, some DSLRs, we’ve absolutely done captures with phones, sorry with using the cameras just the single camera on the phone and using a PC-based photogrammetry processing software, like reality capture. So the quality is great. Absolutely. Now, iPhone has LIDAR scanners and whatnot, we have yet to sort of evaluate if that’s at the quality that we would want it to be, but I can see things certainly going that way.
Shah: I know for a fact that one of our Patreon puzzle was made using a phone that was the door from Prague. That was the one I scanned on my holiday using a Google pixel. And the quality is enough for a hundred piece puzzle. Definitely. As far as the technical side is concerned. There’s obviously the photogrammetry pipeline that things that Reality Capture that are quite process- heavy to create good measures, on our side, the type of what we call segmentation, which is cutting these pieces out is something that happens which requires like quite a lot of PC resources, right? It’s hard to run something like that on the Quest, but who knows what’s going to happen in the future and there is a lot of smart optimization there, that we could do. If we see there is a potential for that, or there is something that it was worthwhile to invest our manpower in.
Az: Just like with VRChat, there’s a lot of very talented people that are taking the time to learn how to make things, whether it’s avatars or with worlds. And so there’s a potential to really create the resources, the documentation, the tutorials for people to create the content that they want. That’s not something we’re ready to do yet. Cause it’s certainly something we need to put a lot of effort into, but I can see us going that way.
Are there other ways to change difficulty?
Ian: People are suggesting, having some kind of Easter eggs hidden in your puzzles, like treasure puzzles you find in completed puzzles, find a puzzle box and you get that puzzle. Another person asking whether you’re going to have multiple difficulty levels for the same level. You obviously have the different piece counts, but is there a way to change the difficulty there?
Az: That is one primary way of changing the difficulty. Twenty-five pieces is exponentially less difficult than 400 pieces. We have other difficulty modifiers. There’s the reference photos that you can either hide or remove, but there’s other things we’re also working on there.
Shah: One of the things that make the puzzles exponentially more difficult is if you take out the traditional shape of the puzzle from the corners of each pieces, we had to add that in mainly because we would also like to play the 400 pieces and playing the 400 pieces without that it’s just extremely frustrating. We are tackling some technical difficulties at the moment or technical challenges, not even difficulties to be able to add that back as a setting.
How did you develop the circular shapes to help fit pieces together?
Ian: I don’t know if you saw our broadcast yesterday we were talking about Puzzling Places, but, have you seen the movie Contact? The way the pages lock together that key part of the story there, I was thinking about that with the actual way some of your pieces connect in the corners, did that take some trial and error to realize you have the little circular shapes that show you how the pieces fit together sometimes? When was that added as a feature and was it hard to get there?
Shah: It was pretty early on. We had the idea, we prototyped it and then created a list of documents to kind of establish, okay, what’s the core game loop. And a key part of the core game loop is to recognize the pattern. And one of the things that we realized is that even in a very good potential puzzle location there are areas where there’s not enough pattern, like the sea, the trees, or whatnot. And these pieces become very frustrating. And if you have a puzzle and 99% of it is amazing, you have details on the walls and you can have fun playing, and there are two pieces that are bad as in, they don’t have enough pattern. You would leave that until the end of the game. And you leave that game with a feeling of negativity because you spent 20 minutes trying to find in which orientation this tree is supposed to go in the other tree. We knew we needed to add a pattern of our own, and that’s where kind of the indentation came in. Our colleague Marcel spent a long time on it. We are quite happy with what came out and I think it really helps the game play to one side on the other side it also makes it a bit easier if these patterns are very strong cues. And it might be the only thing that player might focus on a puzzle where otherwise they would have spend more time paying attention to the textures of the locations. So we have started balancing that for the gameplay a bit, to make the patterns that did the indentations a little bit smaller so that they are not the main dominant thing when you’re looking at a bunch of puzzle pieces.
Ian: Because you’re putting together a 3d object there’s things that you kind of would imagine, go together at a certain angle, but they actually go together quite a different angle. And because it’s a 3d object and you’ve got to have a roof to a building going off at this angle, rather than this angle. And it takes some time to get the right sort of lock between the pieces. But when you have the little indentation there coming off, sometimes it helps you line up those pieces in that three-dimensional way.
Az: I think I know exactly the piece, in fact, you’re thinking about if it’s like in the 25 piece of the Kushiyaki restaurant, the one in Japan, there’s the yellow awnings and then on the edge and the corner, there’s a corner piece of yellow that also has the wall. And we watch a lot of people play this game and they always get the awning correct until that last piece, which has quite a lot of like side wall to it, and they’re like, how does this fit into the yellow? And they realize that, oh yeah, it’s not a flat puzzle, you actually, it goes inwards.
Can you explain the audio design of Puzzling Places?
Ian: Exactly what I was thinking about. You’ve got such playtime with your puzzles to be able to know that sort of thing, kind of like people who know Beat Saber getting played, what song a person is playing just by the way their arms are moving. The more playtesting you can do the better your decisions are about how these things actually work. In the comments one person was also mentioning audio. Let’s talk about the audio design and really share what you were doing there.
Az: The audio is probably one of the biggest parts of the game that we don’t really have a good way of advertising or marketing to people because it’s like, how do you describe immersive soundscapes that play from specific parts of the puzzle? I mean maybe just that way, but the way this actually all came about was when we first were working on the first prototype, the monastery from Armenia, we had recorded ambisonic recordings throughout the entire location. And that’s one of the reasons why we even used it as the puzzle for our prototype, because we had great audio hotspots that once you would finish a part of the puzzle would start to play and you can sort of put your ear against it and hear the life that’s happening in there. And so we sort of progressed on that idea. In fact, we have an incredible audio designer, Pierre-Marie who joined us from Ubisoft. He played the prototype and he said, I love the audio of the game and I was really wanting to help you improve it. And so he came with a lot of ideas, just looking at this reminded me he suggested the idea that the last piece would just be the special thing that has this incredible audio experience around it. You have the final thing you want to put it in. It should feel amazing. And it’s things like that really built on top of each other. The audio is one of my favorite parts of the game, because once you finish the puzzle, or even during it, you can just explore the place by listening to it. And there’s certain sounds also that don’t come from the puzzle, but come from the environment that every time fool me into thinking that they’re coming from outside of my window, they’re like really excellently mixed and mastered that, even with the Quest’s external speakers, they do an incredible job at immersion.
Ian: I love that description of the near final piece where you’re working on the last piece. How recently was that added?
Shah: The idea for the ending was very early on, but we just never had the resources to work on it because also to test the ending of the puzzle, you have to play the puzzle. Right. So every time I wanted to test something and I was like, oh gosh, here it goes. So that’s why it kind of left until the end. And I remember for the beta prototype it was, confetti’s coming off, and it was so unmatching, it didn’t fit anything, not the sound design, nothing, but it just communicated, okay, it’s finished. And we had the idea regarding also the sound design, that there is a philosophy to how you start with kind of nothing. You’ll start with kind of chaos and you put all that together. And the more you put pieces together, the more you learn about the location.
Now, our sound designer, Pierre-Marie. He also completely reflected that in the audio that you start with the bland audio of only the sound of the wind or ambient sound. And then he builds this acoustic landscape. Which is his personal take on the location and that’s what makes it so enjoyable for me because every time I’m experiencing how he would interpret this location. We have these museums and is it like a party going on or children running around? So he recreates his thing and the closer you get to the end you get to that point where now you have a concrete thing. So this last piece is you recreate that whole thing, not in a sense you finished a puzzle, but you got to know the location. You now know how the stairs are connected, where the walls are, where the paintings are, and also the sound. To answer your question, to get to this point, although we had that conceptually to really come up with a concrete plan of how do we do that? This took a year, which is very interesting because right now if I look at the solution we have, I think, well, obviously this is good. Like obviously this is what we should have done from the beginning. But to get here, we did so many bad takes or takes that just didn’t work out. Didn’t communicate. And what we have right now was maybe three weeks before we went gold, the final touches done. Three weeks, something like that. But I’m really happy with that because that was the last pain point for the team for what are we going to do with the end?
Will Puzzling Places support hand tracking?
Ian: How are you guys on hand tracking? Where are you with that and how does that relate to the development of your overall user interface?
Shah: Hand tracking was prototyped internally, we never released that to our testers. At the time it felt very disconnected, when you introduced this embodiment in VR, I think you need to do it right. We want people to get into this meditative flow and everything we designed is there to minimize the amount of noise in our communication with the player and maximize the amount of signal and with hand tracking if you’re doing something and it suddenly disappeared it generates some visual information that doesn’t mean anything to the game. So we love the idea. I love using my headset without my controller, because it’s always hard to find a controller when you put it down. And it’s like, oh God, where was that? At the time it didn’t feel solid enough without us putting a considerable amount of development time in it, but it’s something we’re all interested in. As far as the UX, we did try not to shoot ourselves in the foot and we did try to set up the UX in a way that you can play the game from beginning to the end with a single click. Basically every time we were on a crossroad of design, are we going create some sort of complicated gestures with a controller or some sort of complicated input method to simplify, for expert players a certain action. We decided to split that context to a different tool. So we have this concept of different tools that makes it easier to just use a single click. And with that, I would say hand tracking compared to some other games, should be relatively easy for us to implement if we can sort out the design part, because the game can just be played with a single click
Az: When we first released our prototype, we saw quite a lot of like completely new VR users playing the game and this sort of being a pivotal first VR experience. That’s something that really stuck with us that the entire game can be played with one hand, you don’t even need your second. You could just point and pull pieces or with the same button, grab them and put it together. And that was something really important to never really get rid of. So absolutely that’s the pivotal design principle and maybe hand tracking can fit it.
Ian: We were talking about the different apps that have force- pull actions to reach for objects that are on the other side of the room and target them with a simple action, bring them over. And it seems like that’s a feature that’s going to get a lot better as soon as the headsets have standard eye tracking across the board, games, like I Expect You To Die or even Half-Life: Alyx where you’re looking at objects all around you. It can take a few tries to sometimes grab some of those items. And with so many little pieces all lined up in a row in Puzzling Places, it could be sometimes hard to grab the right piece you want to every time. Are there ways to improve that over time?
Shah: If you have bunch of things next to each other, your brain can distinguish them very easily because they are in different depth layers, the moment you project them on 2d you have the issue that they’re overlapping and the image looks very busy or chaotic. To solve that we started to come up with mechanics like the grouping functionality, so that there are no overlapping pieces. And interestingly enough, we found that also helps with the UX. If we can help the player to organize the environment in a way that pieces do not overlap in depth. That helps a bit with the phenomenon you’re talking about, where they’re going to have an issue. A lot of people play very chaotic. And I do have that the entire time in the back of my mind, how do I improve that so that they have clear speciality, that’s the staircase, that’s the bridge. And there is clear distances between them so that they can easily pick it up. Like designing an environment where people just like cleaning it up. But it’s very difficult. One of the things that I would like to work more and more on so that the players have a more organized environment because it makes the playing experience in my opinion, better. But some people just like to play more chaotic. And I think there’s nothing we can do about that one.
Can you have models with animated portions?
Ian: Asking whether you can have animated parts of the models if you have a stream, you could show an animated stream, with water actually running through it, or whether anything like that as possible, or whether the puzzles sort of need to say static.
Shah: First of all, on PC, almost anything would be possible. One of the main questions we need to answer is how high quality would people like to have their puzzle and how much would they like to pay for it? And that obviously also affects our frequency of releases. If you’re going to focus on a single puzzle for a very long time, we can do amazing things with it. How is the return in terms of the value it creates for the people? Are they going to be exponentially enjoying the puzzle more? Or is it just a tiny bit better for weeks of extra work that you put in?
Az: Do they want to play more puzzles or do they want to play less puzzles that are perhaps more interesting. I’m sure there’s a balance between the two. That’s gonna be our next year for us to figure that out. We learned a lot of things from Patreon. We had a lot of players that loved the Sidequest prototype. All they kept asking for us was more content and they were telling us, we want to give you money. How do we give you money? And we’re like, first of all, thank you. And second, I don’t know, maybe we create a donation box or we also even considered Kickstarter. We were trying to figure out how are we gonna fund this game development up until it’s out. And so Patreon was something that was suggested to us because it was like, it’s not a Kickstarter, there’s a lot of failed Kickstarter, especially with games. Patreon is slightly less committal. You can do incremental sort of releases and people are just there to support you. And so at the time we were essentially testing a lot of puzzles. We made a few. Because of COVID and travel restrictions, it was almost impossible to really get anywhere so we just dug into our own archives, contacted friends, we downloaded and bought things from Sketchfab and, Patreon sort of became this place for us to test ideas, test puzzles that perhaps are not at the really high quality scale as the puzzles that are in the launch, both the puzzle that were in the beta app, as well as the full game, all of these have incredible audio, right? We couldn’t do incredible audio design for every single week of a test puzzle. That’s also the reason why we haven’t released those Patreon puzzles. We didn’t want to have the pressure of having them be so polished that they have to be in the game that are going to be afftecting reviews. That was a sort of a way for us to test things. We have plenty of ideas that we want to create more puzzles and we will find ways of making them more interesting. And this is just definitely the beginning when it comes to what sort of puzzles you will be able to play in VR.
Are you going to be able to show some things at 1:1 scale?
Ian: Scale. Are you going to be able to show some full scale things over time?
Az: We knew this was coming that’s probably like the biggest, biggest ask.
Shah: This is the biggest ask and also internally we discuss this every few months again and again. That’s why when people usually ask, we also don’t give an answer. What I could see potentially happening at some point is if we have one location that we try out at some point, but again, this is not just the technicality of it of can we get this to look good? There’s also a lot of design questions, which the team needs to decide, which we haven’t really got to that decision because every time we decide to talk about it, we say, okay, what’s more important right now? Like the future of Puzzling Places we are really excited about, for example, playing around with multiplayer and stuff like that. Where if given any time of course, or any contractual obligations we have to any given platform, those would come first. We have to decide internally a bit more on that.
Will you support Quest’s AR Passthrough API?
Ian: What about passthrough while we’re on that subject?
Shah: Never tried it.
Az: Yeah. We never tried it. This was also a recent sort of update in the last, what, month from Oculus. So I think it’s funny cause like we spent God, how long did we spend on the environment design of this game? The environment design predates the Sidequest prototype itself, back when we had the SteamVR demos, but I think it technically absolutely can work with pass through. And perhaps that’s also a precursor to like an AR version of Puzzling Places, which it could absolutely work like that. That’s something we’ve talked about a lot is like, okay, when Apple releases something why not have Puzzling Places on it, you don’t need the environment. You just can have the pieces in your room and you can puzzle them together. Perhaps you could do multiple people in your room.
How are sales going?
Ian: You guys are painting this picture of having to focus so much on the task at hand, which is releasing the game for people at home. And you had to prioritize pretty carefully to get everything in place for that launch. Do you have your first sales numbers in and is the VR market as big as you’d hoped?
Shah: So far so good. Yeah. Yeah. So far so good. Like the reviews right now are 130 or something like that, all five out of five. Uh it’s. Yeah. It’s pretty incredible. We are all very happy. I think looking at the initial sales and stuff like that, we are happy with it.
Az: It’s been less than 24 hours. We don’t even have real analytics to look at but even just looking at Patreon, which like this whole Sidequest Patreon, we had in total 1700 people that have supported us on Patreon that downloaded puzzles and played puzzles, having to initially set up like a developer account, even with all that overhead, there was quite a lot of people that were willing to do that to play more puzzles. I think we had 40 to 50,000 downloads of pre-launch of both the Sidequest prototype and the App Lab beta. I think this is gonna appeal to quite a lot of them. I think the bet we’re also making is the players that we have now in VR, a lot of them are gamers, however, VR is becoming cheaper and cheaper and just with the last year or two with the Quest and Quest 2’s launch, we’ve seen the people in our player base completely be a different player base than what we consider to be a VR a user, we have parents, we have grandparents, we have younger people. These were not the hardcore VR people of 2016 and 2017. So that’s probably going to be the type of people that are going to buy VR over the next two to five years. So is Puzzling Places a game for them? We certainly hope so. And that’s certainly how we designed the game to be approachable.
Will you support puzzles like a concert venue that could have an audio cue of a band playing nearby?
Ian: People asking about whether you might support, ads or product placement, like someone paying for a sponsored location to be shown in there. There was also asking about whether you might have a concert venue, with the band playing music nearby?
Shah: That sounds fantastic. Never thought about that, but it sounds fantastic. As a matter of fact, in Berlin, there’s the concert house, which has been scanned and also already cleaned up, it’s an amazing set of models. One of the founders was working archeology. I’ve been painting reconstructing persian palaces since I was 15 for fun, Az is going around the world capturing cultural heritages and the things that he did in Armenia. We are very passionate about historical locations that have a certain story. And one thing we would love to do is use our platform of photogrammetry to bring these locations to people who otherwise would not meet them and to share some of, let’s say, earnings to them. So we have talked about potentially doing things like donations and sharing some revenue with, potentially, cultural heritage sites so that, they can get some money to maintain the location, for the next generation so that they can also see maybe outside of Puzzling Places. Nothing concrete yet.
Az: Photogrammetry and 3d scanning is not all that common everywhere. It is in certain circles, but it’s not like every concert hall has a scan of it or every archeological or cultural heritage sites has it. It’s slowly starting to become more and more popular. Especially with museums starting to digitize their collections. I think Puzzling Places can certainly be an amazing sort of platform and opportunity for us to work more with museums. We’ve focused on places because that’s in our name, Puzzling Places, but we could puzzle quite a lot of things. People have asked us, okay, can we import just 3d models instead of scans? Sure, absolutely. But part of what makes this game special is they are real life things, real life places, and also sometimes real life objects.
Ian: The last comment I’m seeing here says it would be awesome if we could walk through the locations.
Shah: This was the scale again. We can spend five and a half hours talking plus and the minus of walking around. What I can say is if they have access to a Steam realities app launched 2016, I think.
Az: Yes. It was one of the launch apps of SteamVR.
Shah: Yes. Has one-to-one locations where you can walk around, I think we have almost 10 different locations. Yeah. Darn amazing. That can give a feeling of what that could be, if Quest wasn’t Quest and we had more time, but we will see in the future, maybe by Quest 5.
Az: It’s the first ever app that our company created and it’s meant to be like a incredible showcase of photogrammetry quality and exploration, sort of a non-linear experience so that you can go places, see them and learn a little bit more about them. And it’s certainly one of the higher quality photogrammetry things that we’ve done even compared the scans that you’ll see in Puzzling Places. Puzzling Places is the spiritual successor to it because it helps people go places and learn more about them and with just a different game mechanic.
