Developers of generative platforms are adapting to new demands. Enthusiasts and artists are "battling" on social media. And amid all that chaos, new projects based on artificial intelligence are emerging. One of them is Riffusion, which uses an optimized version of Stable Diffusion to create spectrograms, visual representations of sound that we can easily listen to.
Images, videos, voices, music. Artificial intelligence projects continue to evolve… with opinions for and against. Recently we have seen protests, many "exchanges" that amount to digital brickbats, and huge doubts regarding copyright and fair use. Sooner or later, legal conflicts will reach higher instances, but regardless of what happens, artificial intelligence will continue to surprise us.
Today it's the turn of a new generative platform called Riffusion, developed by Seth Forsgren and Hayk Martiros. As its name suggests, Riffusion has the soul of Stable Diffusion, with one difference: the model was specially optimized to generate sonograms/spectrograms. In other words, visual representations of audio based on text prompts.
Riffusion: From Text to Audio with Artificial Intelligence
The page requires no account of any kind, nor the purchase of tokens to generate sounds. In fact, it recommends that users experiment by entering their favorite styles and instruments. Combinations are definitely welcome, for example, "gospel tropical". It is also possible to create prompts that indicate the presence of voices (one of the most interesting I found was "post-teen pop talent show winner"), but we shouldn't expect any clear dialogue.
The sonogram/spectrogram is easy to interpret: the X axis represents time, the Y axis the frequency of sounds, and the color of each pixel its amplitude. The final phase is handled by Torchaudio, which takes the image generated by Stable Diffusion and converts it to audio. The Riffusion About section is excellent and deserves your attention.
Advanced settings allow us to change the image that serves as seed (five different options), and the level of denoising. The higher it is, the more "creative" its output… but it will drift from what you want. This behavior is identical to Stable Diffusion processing images. Another thing it borrows is the weight of prompts. For example, an emphasis on violins can be written as (violin:1.25) in parentheses, or brackets to minimize its impact: [violin] is equivalent to a 1.1x reduction.
Finally, the Riffusion server is being bombarded with requests, so you need to be patient. You will also need good hardware acceleration in your browser, because the official site is quite demanding. Give it a try!
Official site: Click here
