Skip to content

Fix WebSocket reconnection races and Inworld TTS context restore#3591

Open
hwuiwon wants to merge 3 commits intopipecat-ai:mainfrom
hwuiwon:fix-inworld
Open

Fix WebSocket reconnection races and Inworld TTS context restore#3591
hwuiwon wants to merge 3 commits intopipecat-ai:mainfrom
hwuiwon:fix-inworld

Conversation

@hwuiwon
Copy link
Contributor

@hwuiwon hwuiwon commented Jan 29, 2026

This PR improves reliability for websocket-based services during transient disconnects.

  • WebsocketService: serialize reconnect attempts with an asyncio.Lock and short-circuit if the connection is already
    healthy (reduces “reconnect attempt aborted / unable to reconnect” races).
  • InworldTTSService: if a reconnect happens while a context is active, re-send the create message for the existing
    contextId, and only close_context / clear contextId on intentional disconnects (helps prevent “connected but no
    audio / context missing” after reconnects).

@hwuiwon
Copy link
Contributor Author

hwuiwon commented Jan 29, 2026

@markbackman @cshape would appreciate a review!

@codecov
Copy link

codecov bot commented Jan 29, 2026

Codecov Report

❌ Patch coverage is 0% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/pipecat/services/inworld/tts.py 0.00% 7 Missing ⚠️
Files with missing lines Coverage Δ
src/pipecat/services/inworld/tts.py 0.00% <0.00%> (ø)

... and 24 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@markbackman
Copy link
Contributor

@hwuiwon we haven't seen this case. We've also just made changes to the WebsocketService to make it more robust to reconnects. I'm hesitant to make further changes for this release unless we have a documented issue and repro case that occurs on main. If you have that, then we're happy to take a look. I just want to be cautious because changes here affect so many services.

@hwuiwon
Copy link
Contributor Author

hwuiwon commented Jan 30, 2026

@markbackman I’ve been seeing multiple cases like #3538 (comment) while running hundreds of calls through the Inworld WebSocket service. It doesn’t happen consistently, but it reliably shows up under higher call volume. I would suggest making 1000+ calls that uses inworld websocket service to reproduce this. Not so sure how testing is set up for testing this service in particular from pipecat side.

It is critical because this prevents the speech from getting generated and users won't get any feedbacks.

@markbackman
Copy link
Contributor

markbackman commented Jan 30, 2026

@markbackman I’ve been seeing multiple cases like #3538 (comment) while running hundreds of calls through the Inworld WebSocket service. It doesn’t happen consistently, but it reliably shows up under higher call volume. I would suggest making 1000+ calls that uses inworld websocket service to reproduce this. Not so sure how testing is set up for testing this service in particular from pipecat side.

It is critical because this prevents the speech from getting generated and users won't get any feedbacks.

Have you been running from main? Also, is the issue with the Inworld configuration or others? We have customers that do 10K+ calls daily on services like Cartesia and ElevenLabs and they haven't reported issues.

@hwuiwon
Copy link
Contributor Author

hwuiwon commented Jan 30, 2026

I don't believe it's the configuration issue with Inworld because it works normally with the http service. I can rollback the changes in the src/pipecat/services/websocket_service.py but I think we need fix in the src/pipecat/services/inworld/tts.py

@ianbbqzy
Copy link

if a reconnect happens while a context is active, re-send the create message for the existing
contextId

I think this is worth a try. I noticed that Cartesia doesn't have a separate CreateContext message because they expect the configs in every message. On the other hand, 11lab might also not necessitate a createContext message because a lot of configs were already sent as part of query params in the URL.

In the case of Inworld, on Websocket reconnection, it's important to re create a context first

@hwuiwon
Copy link
Contributor Author

hwuiwon commented Feb 3, 2026

@markbackman rolled back the changes in websocket_service.py. Fix only touches Inworld Websocket TTS now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants