Riffusion Colab 🎶

[camenduru]

🎶 Riffusion Colab
https://github.com/camenduru/stable-diffusion-diffusers-colab/blob/main/pytorch_riffusion_stable_diffusion.ipynb

🎶 Riffusion 🤗 Hugging Face Page
https://huggingface.co/riffusion/riffusion-model-v1

[ClashSAN]

anime.mp4

cool, It looks like you'll need to make another extension for playing these..! XD

[enlyth]

I made an extension to generate mp3 files with the model using this repo:

https://github.com/enlyth/sd-webui-riffusion

It's pretty basic so far but works for me

[toyxyz]

Tried with 512x512. Another error occurred.
100%|██████████████████████████████████████████████████████████████████████████████████| 28/28 [00:01<00:00, 15.57it/s] Error completing request:43, 10.84it/s] Arguments: ('blue bird', '', 'None', 'None', 28, 1, False, False, 1, 1, 11, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 0, 0, 9, 0.9, 5, '0.0001', False, 'None', '', 0.1, False, 0, 1, False, False, None, None, '', '', '', '', 'Auto rename', {'label': 'Upload avatars config'}, 'Open outputs directory', 'Export to WebUI style', True, {'label': 'Presets'}, {'label': 'QC preview'}, '', [], 'Select', 'QC scan', 'Show pics', None, False, False, False, False, '', 'Positive', 0, ', ', True, 32, 1, '', 0, '', True, True, False, '', 'None', 30, 4, 0, 0, False, 'None', '<br>', 'None', 30, 4, 0, 0, 4, 0.4, True, 32, '', 5, 24, 12.5, 1000, 'DDIM', 0, 64, 64, '', 64, 7.5, 0.42, 'DDIM', 64, 64, 1, 0, 92, True, True, True, False, False, False, True, 'F:\\Download\\SDWEB_OUTPUT\\mp3', '', False, False, False, False, 'Auto', 0.5, 1, 0, 4, 384, 384, False, False, True, True, True, False, True, 1, False, False, 6.4, 38.5, 0, 0, 384, 384, True, False, True, True, True, False, 1, True, 3, False, 3, False, 3, 1) {} Traceback (most recent call last): File "C:\stable-diffusion-webui\modules\call_queue.py", line 45, in f res = list(func(*args, **kwargs)) File "C:\stable-diffusion-webui\modules\call_queue.py", line 28, in f res = func(*args, **kwargs) File "C:\stable-diffusion-webui\modules\txt2img.py", line 46, in txt2img processed = modules.scripts.scripts_txt2img.run(p, *args) File "C:\stable-diffusion-webui\modules\scripts.py", line 328, in run processed = script.run(p, *script_args) File "C:\stable-diffusion-webui\extensions\sd-webui-riffusion\scripts\riffusion.py", line 47, in run mp3_bytes = self.mp3_bytes_from_wav_bytes(wav_bytes) File "C:\stable-diffusion-webui\extensions\sd-webui-riffusion\scripts\riffusion.py", line 235, in mp3_bytes_from_wav_bytes sound.export(mp3_bytes, format="mp3") File "C:\stable-diffusion-webui\venv\lib\site-packages\pydub\audio_segment.py", line 963, in export p = subprocess.Popen(conversion_command, stdin=devnull, stdout=subprocess.PIPE, stderr=subprocess.PIPE) File "C:\Users\toyxy\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 969, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "C:\Users\toyxy\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1438, in _execute_child hp, ht, pid, tid = _winapi.CreateProcess(executable, args, FileNotFoundError: [WinError 2]

[ClashSAN]

my bad, it must have been hardcoded to 512x512 in the extension (it should produce different height), update the extension in latest commit, there's a fix

[toyxyz]

The same error occurs. FileNotFoundError: [WinError 2]

[enlyth]

@toyxyz I think it's cause you don't have ffmpeg installed. I have to go to sleep, hope this puts you on the right path

[toyxyz]

I installed ffmpeg. now it works thanks!

[ClashSAN]

@enlyth remix ideas from submitted spectograph
some unique interpolation techniques:

• user-submittedimage2user-submittedimage [Feature Request]: Add ability to merge images ad hoc #4455
• prompt instead of seed travel: https://github.com/Kahsolt/stable-diffusion-webui-prompt-travel

There probably are many experts able to assist/are making something cool to use with riffusion, this script by @rewbs looks like the best but I don't know if this can do interpolation or its img2img.
https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts#parameter-sequencer

[enlyth]

@ClashSAN thanks, these are some very helpful resources, I will take a look when I have some time