Skip to content

[Feature] Fully-featured server stop functionality#138

Open
3manifold wants to merge 1 commit intohuggingface:mainfrom
3manifold:stop-server-feature
Open

[Feature] Fully-featured server stop functionality#138
3manifold wants to merge 1 commit intohuggingface:mainfrom
3manifold:stop-server-feature

Conversation

@3manifold
Copy link
Contributor

@3manifold 3manifold commented Nov 12, 2024 •
edited
Loading

Description

This feature implements the server stop functionality during a KeyboardInterrupt in different scenarios.

Although pipeline_manager.stop() seems to be invoked in s2s_pipeline.py during a KeyboardInterrupt, it is observed that in practice stop functionality is unresponsive. Ideally, stop functionality should work in both of the following scenarios (currently not the case):

  • Scenario 1:
    • start server -> stop server
  • Scenario 2:
    • start server -> start client -> stop server

The submitted implementation benefits the application in several aspects:

Tested for for both --device cpu and cuda. Fixes #139 and fixes #140 .

Logs

The logs below demonstrate the output of the server for both scenario 1 and scenario 2 after the fix was applied.

Scenario 1

Click to show log 
python3 s2s_pipeline.py --recv_host 0.0.0.0 --send_host 0.0.0.0 --device cpu --log_level debug 
/home/i00680185/.virtualenvs/voicefixer_main/lib/python3.10/site-packages/df/io.py:9: UserWarning: `torchaudio.backend.common.AudioMetaData` has been moved to `torchaudio.AudioMetaData`. Please update the import path.
...
...
... 12:15:00,827 - utils.thread_manager - DEBUG - Thread 139630012253760 has started. Target: Thread-1 (run), type: <class 'connections.socket_receiver.SocketReceiver'>
... 12:15:00,828 - utils.thread_manager - DEBUG - Thread 139630003861056 has started. Target: Thread-2 (run), type: <class 'connections.socket_sender.SocketSender'>
... 12:15:00,829 - connections.socket_sender - INFO - Sender waiting to be connected...
... 12:15:00,829 - utils.thread_manager - DEBUG - Thread 139629874308672 has started. Target: Thread-3 (run), type: <class 'VAD.vad_handler.VADHandler'>
... 12:15:00,829 - connections.socket_receiver - INFO - Receiver waiting to be connected...
... 12:15:00,830 - utils.thread_manager - DEBUG - Thread 139629865915968 has started. Target: Thread-4 (run), type: <class 'STT.whisper_stt_handler.WhisperSTTHandler'>
... 12:15:00,830 - utils.thread_manager - DEBUG - Thread 139629481412160 has started. Target: Thread-5 (run), type: <class 'LLM.language_model.LanguageModelHandler'>
... 12:15:00,831 - utils.thread_manager - DEBUG - Thread 139629473019456 has started. Target: Thread-6 (run), type: <class 'TTS.parler_handler.ParlerTTSHandler'>
... 12:15:00,831 - utils.thread_manager - DEBUG - Thread 139630012253760 attempt to join. Target: Thread-1 (run)
Stopping server..
Server closed.
... 12:15:44,945 - utils.thread_manager - DEBUG - Server stop was invoked
... 12:15:44,945 - connections.socket_receiver - DEBUG - Receiver is in stopping process. Sending END message to the next component to initiate server termination..
... 12:15:44,945 - connections.socket_sender - DEBUG - SocketSender: shutdown socket
... 12:15:44,945 - baseHandler - DEBUG - VADHandler: Received END message, stopping thread
... 12:15:44,945 - connections.socket_sender - DEBUG - SocketSender received exception: [Errno 22] Invalid argument. Possibly the sever is in termination process..
... 12:15:44,946 - connections.socket_sender - INFO - Sender closed
... 12:15:44,946 - baseHandler - DEBUG - VADHandler: Stopping pipeline component..
... 12:15:44,946 - baseHandler - DEBUG - WhisperSTTHandler: Stopping pipeline component..
... 12:15:44,946 - baseHandler - DEBUG - LanguageModelHandler: Stopping pipeline component..
... 12:15:44,946 - baseHandler - DEBUG - ParlerTTSHandler: Stopping pipeline component..
... 12:15:44,946 - utils.thread_manager - DEBUG - Thread 139630012253760 attempt to join. Target: Thread-1 (run)
... 12:15:44,946 - utils.thread_manager - DEBUG - Thread 139630012253760 has joined. Target: Thread-1 (run)
... 12:15:44,946 - utils.thread_manager - DEBUG - Thread 139630003861056 attempt to join. Target: Thread-2 (run)
... 12:15:44,946 - utils.thread_manager - DEBUG - Thread 139630003861056 has joined. Target: Thread-2 (run)
... 12:15:44,946 - utils.thread_manager - DEBUG - Thread 139629874308672 attempt to join. Target: Thread-3 (run)
... 12:15:44,946 - baseHandler - DEBUG - WhisperSTTHandler: Received END message, stopping thread
... 12:15:44,946 - utils.thread_manager - DEBUG - Thread 139629874308672 has joined. Target: Thread-3 (run)
... 12:15:44,947 - baseHandler - DEBUG - LanguageModelHandler: Received END message, stopping thread
... 12:15:44,947 - utils.thread_manager - DEBUG - Thread 139629865915968 attempt to join. Target: Thread-4 (run)
... 12:15:44,947 - utils.thread_manager - DEBUG - Thread 139629865915968 has joined. Target: Thread-4 (run)
... 12:15:44,947 - utils.thread_manager - DEBUG - Thread 139629481412160 attempt to join. Target: Thread-5 (run)
... 12:15:44,947 - utils.thread_manager - DEBUG - Thread 139629481412160 has joined. Target: Thread-5 (run)
... 12:15:44,947 - utils.thread_manager - DEBUG - Thread 139629473019456 attempt to join. Target: Thread-6 (run)
... 12:15:44,947 - baseHandler - DEBUG - ParlerTTSHandler: Received END message, stopping thread
... 12:15:44,947 - utils.thread_manager - DEBUG - Thread 139629473019456 has joined. Target: Thread-6 (run)

Process finished with exit code 0

Scenario 2

Click to show log 
python3 s2s_pipeline.py --recv_host 0.0.0.0 --send_host 0.0.0.0 --log_level debug 
...
...
... 11:59:26,387 - utils.thread_manager - DEBUG - Thread 140596932494912 has started. Target: Thread-1 (run), type: <class 'connections.socket_receiver.SocketReceiver'>
... 11:59:26,389 - utils.thread_manager - DEBUG - Thread 140596924102208 has started. Target: Thread-2 (run), type: <class 'connections.socket_sender.SocketSender'>
... 11:59:26,390 - connections.socket_sender - INFO - Sender waiting to be connected...
... 11:59:26,390 - connections.socket_receiver - INFO - Receiver waiting to be connected...
... 11:59:26,393 - utils.thread_manager - DEBUG - Thread 140596915709504 has started. Target: Thread-3 (run), type: <class 'VAD.vad_handler.VADHandler'>
... 11:59:26,395 - utils.thread_manager - DEBUG - Thread 140596907316800 has started. Target: Thread-4 (run), type: <class 'STT.whisper_stt_handler.WhisperSTTHandler'>
... 11:59:26,398 - utils.thread_manager - DEBUG - Thread 140596778821184 has started. Target: Thread-5 (run), type: <class 'LLM.language_model.LanguageModelHandler'>
... 11:59:26,400 - utils.thread_manager - DEBUG - Thread 140596770428480 has started. Target: Thread-6 (run), type: <class 'TTS.parler_handler.ParlerTTSHandler'>
... 11:59:26,401 - utils.thread_manager - DEBUG - Thread 140596932494912 attempt to join. Target: Thread-1 (run)
... 11:59:47,059 - connections.socket_receiver - INFO - Receiver connected
... 11:59:47,060 - connections.socket_sender - INFO - Sender connected
... 11:59:51,812 - VAD.vad_handler - DEBUG - VAD: end of speech detected
... 11:59:51,815 - VAD.vad_handler - DEBUG - Stop listening
... 11:59:51,816 - baseHandler - DEBUG - VADHandler:  0.004 s
... 11:59:51,816 - STT.whisper_stt_handler - DEBUG - infering whisper...
... 11:59:51,878 - STT.whisper_stt_handler - DEBUG - finished whisper inference
... 11:59:51,912 - STT.whisper_stt_handler - DEBUG - Language Code Whisper: a
... 11:59:51,912 - baseHandler - DEBUG - WhisperSTTHandler:  0.096 s
... 11:59:51,915 - LLM.language_model - DEBUG - infering language model...
USER: � fakeiron�
...
...
... 12:00:12,585 - baseHandler - DEBUG - LanguageModelHandler:  20.673 s
ASSISTANT: I'm glad you like it!
... 12:00:16,100 - baseHandler - DEBUG - LanguageModelHandler:  3.515 s
... 12:00:18,021 - parler_tts.modeling_parler_tts - WARNING - `prompt_attention_mask` is specified but `attention_mask` is not. A full `attention_mask` will be created. Make sure this is the intended behaviour.
... 12:00:21,834 - baseHandler - DEBUG - LanguageModelHandler:  5.733 s
... 12:00:38,372 - utils.thread_manager - DEBUG - Server stop was invoked
... 12:00:38,372 - connections.socket_receiver - DEBUG - Receiver is in stopping process. Sending END message to the next component to initiate server termination..
... 12:00:38,373 - connections.socket_sender - DEBUG - SocketSender: shutdown socket
... 12:00:38,373 - baseHandler - DEBUG - VADHandler: Received END message, stopping thread
... 12:00:38,373 - baseHandler - DEBUG - VADHandler: Stopping pipeline component..
... 12:00:38,374 - baseHandler - DEBUG - WhisperSTTHandler: Stopping pipeline component..
... 12:00:38,375 - baseHandler - DEBUG - LanguageModelHandler: Stopping pipeline component..
... 12:00:38,375 - baseHandler - DEBUG - ParlerTTSHandler: Stopping pipeline component..
... 12:00:38,375 - utils.thread_manager - DEBUG - Thread 140596932494912 attempt to join. Target: Thread-1 (run)
... 12:00:38,376 - utils.thread_manager - DEBUG - Thread 140596932494912 has joined. Target: Thread-1 (run)
... 12:00:38,376 - baseHandler - DEBUG - WhisperSTTHandler: Received END message, stopping thread
... 12:00:38,376 - utils.thread_manager - DEBUG - Thread 140596924102208 attempt to join. Target: Thread-2 (run)
... 12:00:38,377 - baseHandler - DEBUG - LanguageModelHandler: Received END message, stopping thread
Stopping server..
... 12:00:38,428 - connections.socket_receiver - INFO - Receiver closed
... 12:03:09,340 - baseHandler - DEBUG - ParlerTTSHandler:  176.754 s
... 12:03:09,342 - connections.socket_sender - INFO - Sender closed
... 12:03:09,343 - utils.thread_manager - DEBUG - Thread 140596924102208 has joined. Target: Thread-2 (run)
... 12:03:09,343 - utils.thread_manager - DEBUG - Thread 140596915709504 attempt to join. Target: Thread-3 (run)
... 12:03:09,343 - utils.thread_manager - DEBUG - Thread 140596915709504 has joined. Target: Thread-3 (run)
... 12:03:09,343 - utils.thread_manager - DEBUG - Thread 140596907316800 attempt to join. Target: Thread-4 (run)
... 12:03:09,343 - utils.thread_manager - DEBUG - Thread 140596907316800 has joined. Target: Thread-4 (run)
... 12:03:09,343 - utils.thread_manager - DEBUG - Thread 140596778821184 attempt to join. Target: Thread-5 (run)
... 12:03:09,343 - utils.thread_manager - DEBUG - Thread 140596778821184 has joined. Target: Thread-5 (run)
... 12:03:09,343 - utils.thread_manager - DEBUG - Thread 140596770428480 attempt to join. Target: Thread-6 (run)
Server closed.
... 12:06:42,062 - baseHandler - DEBUG - ParlerTTSHandler:  212.720 s
... 12:06:42,063 - utils.thread_manager - DEBUG - Thread 140596770428480 has joined. Target: Thread-6 (run)

Process finished with exit code 0

This was referenced Nov 12, 2024
Open
Open
@3manifold 3manifold marked this pull request as ready for review November 12, 2024 14:16
@Uldiniad
Copy link

Uldiniad commented Jun 23, 2025

I cherry-picked the commit on top of latest on main branch. It crashed unless I implemented the following function:

diff --git a/connections/local_audio_streamer.py b/connections/local_audio_streamer.py
index 389dcb8..7666cfd 100644
--- a/connections/local_audio_streamer.py
+++ b/connections/local_audio_streamer.py
@@ -42,3 +42,8 @@ class LocalAudioStreamer:
             while not self.stop_event.is_set():
                 time.sleep(0.001)
             print("Stopping recording")
+
+    def stop(self):
+        """Stops the audio stream by setting the stop event."""
+        logger.info("Stopping local audio stream")
+        self.stop_event.set()

@Uldiniad
Copy link

Uldiniad commented Jun 23, 2025 •
edited
Loading

I cherry-picked the commit on top of latest on main branch. It crashed unless I implemented the following function:

diff --git a/connections/local_audio_streamer.py b/connections/local_audio_streamer.py
index 389dcb8..7666cfd 100644
--- a/connections/local_audio_streamer.py
+++ b/connections/local_audio_streamer.py
@@ -42,3 +42,8 @@ class LocalAudioStreamer:
             while not self.stop_event.is_set():
                 time.sleep(0.001)
             print("Stopping recording")
+
+    def stop(self):
+        """Stops the audio stream by setting the stop event."""
+        logger.info("Stopping local audio stream")
+        self.stop_event.set()

even with the above, I then get

File "/usr/lib/python3.12/threading.py", line 1151, in join
    self._wait_for_tstate_lock(timeout=max(timeout, 0))
File "/usr/lib/python3.12/threading.py", line 1167, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):

coming from

s2s_pipeline.py", line 483, in main
    pipeline_manager.stop()

Edit: this seems like it could be related to wsl audio setup. Disabling systemd-timesyncd now I see

^CStopping server..
2025-06-23 14:25:11,886 - connections.local_audio_streamer - INFO - Stopping local audio stream
Stopping recording
Server closed.

but it still hangs (doesn't fully exit)

@3manifold
Copy link
Contributor Author

3manifold commented Nov 14, 2025

@Uldiniad Thank you for your comment. There were several changes merged on the main branch after this feature was implemented. I'll allocate some time to rebase and test the functionality using the latest main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] Server stop functionality is unresponsive [Bug] Unhandled exceptions emerging from orphan threads on server

2 participants

@3manifold @Uldiniad