Introduction
Imagine having a personal accompanist that completes your piano pieces in real-time, directly from your phone. That's exactly what Simon Edwardsson achieved by training a 125-million parameter model to autocomplete piano performances through an app called RollTab. Inspired by GitHub Copilot but for music, this project took nearly a year of development and optimization.
The Magic of MIDI Files
To understand how this tool works, it's essential to grasp what a MIDI file is. Unlike traditional audio formats like MP3, a MIDI file encodes music as a series of events: a key is pressed at a certain pitch and velocity, a key is released, the sustain pedal changes state, etc. These events are organized into multiple tracks, allowing for rich musical diversity.
Tokenizing Musical Events
One of the major challenges in developing this model was converting MIDI events into a discrete sequence that the model could read and predict. Instead of creating a token for every unique event, Simon opted for a factored approach using a reduced grammar. For instance, instead of having a vast combination of NOTE_ON and NOTE_OFF for each pitch and velocity, he structured the tokens as [NOTE_ON, PITCH, VELOCITY], significantly reducing the vocabulary size and easing the model's learning process.
Training the Model
The model, based on a 125M parameter transformer, was trained on cleaned and optimized MIDI data. The aggressiveness in data cleaning and selecting the right MIDI representation were critical to improving the model's performance. Additionally, a technique called DPO (Diverse Penalization Optimization) was added post-training to further refine the prediction quality.
Real-time Performance
One of the most impressive aspects of this project is its ability to run in real-time on an iPhone 15, producing around 108 notes per second. This opens up incredible possibilities for musicians, whether amateur or professional, to collaborate with AI and explore new creative dimensions.
Conclusion
The ability of an AI model to autocomplete musical performances in real-time on a mobile device marks a significant advancement in human-machine interaction. It could transform how we create and play music, making composition accessible to everyone. If you're curious about how this technology can be applied to your project, let's discuss it in 15 minutes.
Let's discuss your project in 15 minutes.
Feel free to contact us to delve deeper into this fascinating subject and explore how this technology can be integrated into your musical environment.