Late last month, Chris Donahue and Abhay Agarwal launched an impressive new tool called ‘Beat Sage’, which uses artificial intelligence and neural networks to automatically generate custom Beat Saber maps from any song in seconds.
We tried it out, and were left suitably impressed — the resulting tracks are fun, challenging and better than many other auto-map generators for rhythm games. With the tool still in active development, we reached out to Donahue and Agarwal to get a better understanding of what makes Beat Sage tick and how it might be improved in the future.
Getting Started
“I first tried Beat Saber in December 2019 and loved it immediately,” Donahue, an AI researcher at Stanford University, explained in an email. A month later, in January 2020, he began intermittent work on what would become Beat Sage and spoke to Agarwal, who runs an AI design firm called Polytopal and agreed to help out. The two of them worked on Beat Sage in their spare time for the next few months and launched in April.
Since launch they’ve added support for bombs and the ability to generate maps from MP3 uploads. Their next update will deliver a set of much anticipated features, including one saber maps, obstacles and support for 90 degree map generation.
The pair reiterate that their tool remains in active development with more features and improvements to the AI still to come. Beat Sage may not be perfect, but it does appear to be one of the better automatic map generators available for any rhythm game, let alone Beat Saber. At its best, it produces AI-generated maps that are engaging, nuanced and sometimes even challenging.
The Secret to AI Success
So why does Beat Sage work so well? Well, it uses an AI that has been trained on human-made maps and applies that to neural networks, allowing it to develop a custom map for any piece of music. To be clear, this isn’t just a computer determining the beats per minute (BPM) of an audio track and assigning notes and patterns that align with the beat.
Instead, Agarwal and Donahue curated a sample of select official and community-made maps which the AI used to teach itself how Beat Saber maps are constructed. It essentially tried to teach itself what works well for certain types of audio, music and instruments, based of the selection it was given.
“Beat Sage is looking at the audio spectrogram and learning (from data) to answer the question ‘how likely would it be for a human to put a block at this point in time?'” the creators explained in an email.
The resulting AI is completely self-taught and can generate a map from any audio track.
Genre Bias
However, because the model’s sample tracks mainly feature fast electronic-based music, the AI does skew itself to work better for certain types of songs. Certain genres, such as pop and electronic music, often result in better generated maps, thanks to the similarities they share with the model tracks. Other genres, such as rock music or songs with a slow tempo or sparse instrumentation, can be a real mixed bag — the result can range from entertaining to inconsistent to just boring or even fundamentally broken.
“We are aware that Beat Sage does better on some genres than others,” they explained. “For the time being, we’re reasonably happy with how the audio analysis system performs overall. Improving the audio analysis is definitely on the roadmap though.”
Donahue explained that Beat Sage “shines brightest on tracks which have faster beats and a wide variety of rhythm changes. Songs with rhythms that change frequently are more difficult for humans to map, but because Beat Sage doesn’t depend as much on tempo as humans, it tends to excel at this category of songs.”
A consistent music tempo, or BPM, is relatively unimportant when Beat Sage generates a map. In fact, the tool doesn’t really take overall BPM into consideration — instead, it looks at each slice of music individually. “It only looks at a small window of the spectrogram,” they explained. “It focuses less on tempo and more on the instantaneous presence of percussive instruments.”
A Human Touch
Beat Sage tracks often surprise you with nuanced moments that feel eerily… human. The AI can pick up on significant moments — such as a drum fill, a big vocal moment, or a big drop — and turns them into fun moments for you to play through. If a human-made map uses beat blocks to emulate a drum fill, Beat Sage will learn to integrate similar moments its own tracks.
“We didn’t program this behavior. It comes from the aforementioned tendency of Beat Sage to place blocks based on the presence (or lack thereof) of instruments as opposed to the underlying tempo,” the creators explained.
“All of these moments you mention are shocking/cool primarily because they’ve likely been “discovered” by humans first. We’re a long way away from AI learning to do these kinds of things without any training data at all. The AI would probably have to know a lot more about humans in general, what interests us, how our emotions work, how we respond intrinsically to music, etc.”
Coming Soon
Beat Sage’s next update doesn’t feature any huge overhauls to the AI system, except for a slightly “redesigned training pipeline.” The selection of curated maps used by the AI has also been slightly altered to exclude some that took advantage of non-standard game modifications that they weren’t initially aware of. “The result is that the newer model seems to be doing a bit better than our previous one on quantitative metrics. Whether this will result in a perceivable change in map quality remains to be seen.”
Donahue and Agarwal expressed interest in building the tool into a fully-fledged game modification, that could allow users to instantly generate maps from within Beat Saber itself and instantly give feedback on each map after playing it. They also plan to implement “guard rails” into the AI, which will stop it from generating blocks and patterns that lead to undesirable experiences like handclaps.
For now, the biggest roadblock is time. While Beat Sage does have a Patreon page, the tool is still very much a side project.
“We’ll see where it goes. We’re both gainfully employed, so we don’t have any immediate need to earn a living off of Beat Sage … That said, if anyone has any cool problems that they think this tech could solve, definitely reach out to us!”
You can try Beat Sage out for yourself now, either via a YouTube link or an MP3 upload. To learn how to install any custom Beat Saber map, from Beat Sage or elsewhere, check out our guide here.
