riffusion

Riffusion : Stable diffusion for real-time music generation

Input

Prompt

• Example

jazzy rapping from paris

Output

output.mov

Requirements

This model requires additional module.

pip3 install torch
pip3 install torchaudio
pip3 install transformers
pip3 install soundfile

This model can use optional modules.

pip3 install pydub

Usage

Automatically downloads the onnx and prototxt files on the first run. It is necessary to be connected to the Internet while downloading.

For the sample image,

$ python3 riffusion.py

If you want to specify the input text, put the text after the --input option.
You can use --savepath option to change the name of the output file to save.

$ python3 riffusion.py --input TEXT --savepath SAVE_WAV_PATH

Reference

• Riffusion
• Hugging Face - Riffusion

Framework

Pytorch

Model Format

ONNX opset=14

Netron

unet.onnx.prototxt
text_encoder.onnx.prototxt
vae_decoder.onnx.prototxt