AI 'augmented artists': How Sony CSL and Whim Therapy are creating the music of the future

Music, its creation as its industry, are not long calm rivers. Left exhausted by the revolution of MP3 and digital formats, trying to rebuild themselves in particular around streaming in which they have placed a few marbles, the scalded record majors fear a cold future - and are now trying to regain a head start on the technology, to shape a future over which they will not have lost control this time.

At Sony, it is within Sony CSL – for “Computer Science Laboratories” – that these new horizons are sought and invented. "Sony CSL was founded in 1988 in Tokyo by the inventor of the CD, who saw that technological advances were accelerating and having more and more impact on the world", explains Michael Turbot, Technology promotion manager of the Parisian branch of the structure.

"He wanted to set up a structure that was independent of Sony's core business, which would focus on understanding all new technologies, regardless of business areas, and making sure that Sony was going to use them in a positive way for the world and the planet,” he continues.

"All of my colleagues at Sony CSL are doctors, researchers and engineers, so a big part of my job is to understand and then translate scientific discussions or reflections into a language suitable for the music industry, and succeed in bringing them into in artistic reflections or strategies at Sony Music or Sony Music Publishing.”

“In Paris, there are four areas of activity, mainly based on artificial intelligence: language, creativity, sustainable development and a large team dedicated to research on AI and understanding its contribution to the creative process of the artists and producers of tomorrow.”

Does that mean that the hits spinning in your eardrums today are already written by robots? Not quite – and it probably never will be.

But things are progressing at their own pace, and the impact of this research is beginning to be understood. “We are entering a phase where all our prototypes are sufficiently manageable and are beginning to be used concretely by artists. All this takes a lot of time and we didn't want to communicate until we were sure that what we were doing was bringing real added value in the studio for the artists.

This is now the case. “Producer Twenty9 released last December the first drumkit in which all sounds were generated by our AIs, before he reworked on it. A lot of songs released recently may therefore contain sounds from our lab, without us knowing it,” says Michael Turbot.

“We constantly collaborate with artists so that they test our prototypes and that we can make sure that they are useful to them. This is what is being done at the moment with Jean-Michel Jarre, Niro, Uele Lamore, Twenty9 and others.

Among these others, there is notably Whim Therapy. Sony CSL, which in its technological clearing work is joined by structures such as Google Magenta, Open AI, Spotify or Ircam, is presenting one of the pieces from Le Parisien at the 2021 AI Song Contest, a sort of Eurovision music assisted by artificial intelligence.

Whim Therapy answered some of our questions about the genesis of this “Let it go”, for which you can vote at this address, and about the way in which machines and algorithms mingled with its soul to give birth to the piece.

korii.: What was your first approach to artificial intelligence? What was your vision of the thing, before being able to explore it concretely yourself?

Whim Therapy: It all started with a nightly Instagram scroll. I came across the Sony CSL page, which explained developing AI tools for artists.

Intrigued, I contacted them through Jean-Christophe Bourgeois, my editor at Sony Music Publishing in France, and they invited me to their lab so that I could discover their technology, before lending it to me so that I could can experience it. I felt like I was stepping into the future, having access to something that will be everywhere in twenty years, it was quite exciting.

What can you tell me about the tools created by Sony CSL? How can a composer approach them, how can he use them?

CSL tends to refer to artists who use their technology as "augmented artists". It made me laugh at first, but in the end I believe they are right. The more I use the tools, the more I realize their contribution, especially in the ability to unlock the composer, thanks to the proposals of the AI.

It requires adapting the creative process a bit. Instead of deciding everything on my own, I let the AI ​​make guesses and try to figure out what it's trying to suggest.

You could imagine him as a pal in the room, who has difficulty articulating his ideas but who has a particularly creative spirit. The sound generation tools are also great for looking for unique colors.

What does the music of the future look like? A little about that – the interfaces of DrumNet and NOTONO, two of the tools offered by Sony CSL. | sony

More specifically, what can you tell me about Impact Drums, DrumGan, BassNet, DrumNet, Poiesis, Resonance EQ and NOTONO, the Sony CSL tools you used?

Impact Drums and DrumGan are quite similar, both are drum sound generators, with endless possibilities: it's royal to instantly have unique sounds.

BassNet and DrumNet are pattern and melody generators, one creating bass lines and the other drum parts. There are a few knobs to turn to change how the AI ​​will approach the music, such as the complexity of the generated part or the note durations.

I find them interesting, in particular because they will bring very different ideas from those I may have, nourished by my automatisms. This makes it possible to take the pieces in new directions or give them new colors.

Poiesis is an assistant for an author: he reads and understands the text that is provided to him, and comes to propose other possibilities on the parts of the text where I block. It's a huge time saver to have ideas out of the box, generated in one click. Although frankly, Poiesis likes to bring back themes of money, drugs or sex, he is a great writing partner.

Resonance EQ is a resonant frequency attenuator, which I find particularly interesting when used upside down: it brings a lot of grain and imperfections to the sound, and therefore character.

As for NOTONO, it is a sound generator via spectrogram. We mix timbres of different instruments on each zone of the harmonic spectrum, and we decide on the duration of the notes to generate unique sounds. I use it to create samples, which are then transformed into instruments and which bring a particular texture to the songs.

How is the work of the human and that of the machine articulated in the case of writing a piece like “Let it go”? You talk about a functioning similar to that of a group...

On "Let it go", I started with a chord chart and an intention, just to leave as much room as possible for the AI. There was a first studio session, where we worked with my guys on a beginning of text, topliner, arrangements, and where we recorded lots of instruments to feed the AI.

I then used DrumGan, Impact Drums and NOTONO to create the drum, bass and synthesizer sounds that are heard in the track. Then I used DrumNet and BassNet to propose things.

Afterwards, I used the sounds created previously to play the generated patterns, and that's when the song took an unexpected turn, especially in its second part. The group side arrived, thanks to an input of proposals that I probably could not have had alone – the influences of the AI ​​are very different from mine.

Once the instrumental part was finished, I finished the text before reworking it with Poiesis. To be honest, some of the coolest lines on the track come from him.

What do you think AI can bring, in the long term, to the world of music and art in general? Can AI be limiting or, conversely, can it pave the way for whole new things, and how?

I have a feeling that AI will add strings to an artist's bow, hence this notion of "augmented artists". His ability to propose things whatever the track we are working on is an important contribution. For the moment, it requires the artist to adapt his workflow to it, limiting himself to the passage, but the balance will probably be able to reverse over the years.

I have a real curiosity about the artists of the future, who will be able to start with this technology already available. No longer needing to be as good at the things the AI ​​can create with them, they can focus on what I think advances art the most: ideas.

What can we say to people who might be afraid of a certain “dehumanization” of creation, or of creation aimed too directly at success through the analysis of data?

No stress, we are far from having the magic button that creates a tube. AI only comes up with ideas based on what we feed it with, so without artists, there will be nothing. And AI doesn't have the ability to tell a good idea from a bad one.

It may well be able to propose interesting evolutions to a song, it is the composers and authors who always have the last word. There is no recipe for a good song. There was none before AI, and there still is. I think it's best to approach it as an evolution of the tools available to create.