Enable GPU support

[mreilaender]

Enables GPU support by adding --cuda to the argument parser and passing it to piper

Closes #1

[edurenye]

Looks good to me, and this would be a great thing to have.
Can you please review it @synesthesiam ?

[synesthesiam]

The piper executable does not support the --cuda flag

[edurenye]

Hi @synesthesiam, thank you for your quick answer.

According to the documentation from here: https://github.com/rhasspy/piper (line https://github.com/rhasspy/piper/blob/6c5e283439f8400aa7a2652aafbedfb77ca3fc22/README.md?plain=1#L176) piper has a --cuda flag.

Is it a bug there that cuda does not work (I see you just commited something in that repo related to cuda) or is it in this other issue rhasspy/rhasspy3#49?

Can we help you in some way?

[synesthesiam]

The Python library for piper supports CUDA. I added the flag for the C++ version yesterday, but it needs the "gpu" version of the onnx runtime so I will need to adjust the build process too.

[ashald]

@synesthesiam is there a way to leverage the latest C++ version that has the --cuda parameter? Thanks!

[ms1design]

+1, @synesthesiam I would be appreciated if you could guide us a bit how to utilise cuda in our Jetson project here (using cuda12.2 and onnxruntime-gpu): dusty-nv/jetson-containers#481

Edit: Solved here: dusty-nv/jetson-containers@4ba5dc4

[roryeckel]

I agree with @synesthesiam, piper supports "--use-cuda", not "--cuda"

Source: https://github.com/rhasspy/piper/blob/c0670df63daf07070c9be36b5c4bed270ad72383/src/cpp/main.cpp#L441

Please see my comment on a related project where I encountered the same issue here: linuxserver/docker-piper#8 (comment)

[edurenye]

Good catch @roryeckel !

Can you @mreilaender update this PR please?

[edurenye]

Seems like after using --use-cuda let's us move a bit forward, but then I encounter this error:

wyoming-piper-1    | ERROR:asyncio:Task exception was never retrieved
wyoming-piper-1    | future: <Task finished name='wyoming event handler' coro=<AsyncEventHandler.run() done, defined at /opt/venv/lib/python3.11/site-packages/wyoming/server.py:31> exception=FileNotFoundError(2, 'No such file or directory')>
wyoming-piper-1    | Traceback (most recent call last):
wyoming-piper-1    |   File "/opt/venv/lib/python3.11/site-packages/wyoming/server.py", line 41, in run
wyoming-piper-1    |     if not (await self.handle_event(event)):
wyoming-piper-1    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
wyoming-piper-1    |   File "/opt/venv/lib/python3.11/site-packages/wyoming_piper/handler.py", line 53, in handle_event
wyoming-piper-1    |     raise err
wyoming-piper-1    |   File "/opt/venv/lib/python3.11/site-packages/wyoming_piper/handler.py", line 48, in handle_event
wyoming-piper-1    |     return await self._handle_event(event)
wyoming-piper-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
wyoming-piper-1    |   File "/opt/venv/lib/python3.11/site-packages/wyoming_piper/handler.py", line 108, in _handle_event
wyoming-piper-1    |     wav_file: wave.Wave_read = wave.open(output_path, "rb")
wyoming-piper-1    |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
wyoming-piper-1    |   File "/usr/lib/python3.11/wave.py", line 630, in open
wyoming-piper-1    |     return Wave_read(f)
wyoming-piper-1    |            ^^^^^^^^^^^^
wyoming-piper-1    |   File "/usr/lib/python3.11/wave.py", line 280, in __init__
wyoming-piper-1    |     f = builtins.open(f, 'rb')
wyoming-piper-1    |         ^^^^^^^^^^^^^^^^^^^^^^
wyoming-piper-1    | FileNotFoundError: [Errno 2] No such file or directory: ''

This issue was already reported in this other issue: OHF-Voice/wyoming#9

[roryeckel]

Indeed I experienced the same issue in linuxserver/docker-piper#8 (comment) , it seems to be a segfault

[edurenye]

Why do you think it might be a segfault? I was thinking more about a permissions issue, since the error says: "No such file or directory"
But I have no idea

[roryeckel]

Why do you think it might be a segfault? I was thinking more about a permissions issue, since the error says: "No such file or directory"
But I have no idea

I came to that conclusion by running the piper binary inside the container without the python wyoming scaffolding around it. The reason it fails to load a file is because empty string is returned by piper, which is interpreted as a filename. Certainly empty string is not a valid file. But the segfault is the reason for empty string.

[ms1design]

Piper stores the wavefile on filesystem. It's a permissions issue. Piper works perfectly with CUDA and onnxruntime-gpu in jetson-containers: https://github.com/dusty-nv/jetson-containers/blob/master/packages/smart-home/wyoming/piper/Dockerfile

You can find a git patch there, showcasing fow to fix cuda flag issues: https://github.com/dusty-nv/jetson-containers/blob/master/packages/smart-home/wyoming/piper/wyoming-piper_cuda_path.diff

Cheers.

[mreilaender]

@edurenye done. I used pull --rebase I hope that's fine. Let me know if it isn't

[edurenye]

@mreilaender Thanks for updating the PR. The rebase looks fine, but is not enough changing the input argument, what you really need to change is the past argument, so you changed the __main__.py file, but missed changing the process.py file. See what I did here: rhasspy/wyoming-addons@4da560b

[mreilaender]

Done :)

[edurenye]

Thanks! It looks good to me now

[fti7]

Any Updates here? 👀

[RaymondChen299]

Hi guys, I am excited to see this feature comming. Is there any timeline on the next compile on the offical docker?

[lerra]

Same here, looking forward to the feature and thanks big times for the effort on this project, it's really awesome

[mreilaender]

@synesthesiam what's the status here? Do you need something else from me or anyone else?

[OCT0PUSCRIME]

I've been testing this the last couple days and finally got it figured out after testing and troubleshooting. There seems to be some misinformation out there so here is what I have found:

• The preshipped piper binary does not support --cuda or --use-cuda
• You can run piper via python or a binary - wyoming is kind of weird because it's a python wrapper but it still uses the binary - I could not get it to work with piper using python, which is a bummer because GPU is easier to get working with piper python.
• To get a binary with CUDA support you have to clone the piper repo and build it yourself. (read piper repo notes on this for the phonemize caveat)
• Once the binary is built with CUDA support it still won't work. It will say it's missing if you try to invoke --use-cuda libonnxruntime_providers_cuda.so and libonnxruntime_providers_shared.so
  to fix this run:

wget https://github.com/microsoft/onnxruntime/releases/download/v1.14.1/onnxruntime-linux-x64-gpu-1.14.1.tgz

tar -xzf onnxruntime-linux-x64-gpu-1.14.1.tgz

sudo cp onnxruntime-linux-x64-gpu-1.14.1/lib/libonnxruntime*.so /usr/share/piper/

This last command will overwrite some files that need to be overwritten for this to work.

[synesthesiam]

My plan is to migrate this repo to using the Python version of Piper once the piper1-gpl repo is ready: https://github.com/OHF-Voice/piper1-gpl/

[Rudd-O]

Can this be polished to a minimum so only the Python changes are merged? We would really like to use accelerated TTS generation here at home.

[RyanMorash]

My plan is to migrate this repo to using the Python version of Piper once the piper1-gpl repo is ready: https://github.com/OHF-Voice/piper1-gpl/

Is this resolved now that piper1-gpl is used as of 2.0.0