Skip to content

TTS services improvements.#3584

Open
filipi87 wants to merge 23 commits intomainfrom
filipi/speak_frame
Open

TTS services improvements.#3584
filipi87 wants to merge 23 commits intomainfrom
filipi/speak_frame

Conversation

@filipi87
Copy link
Contributor

@filipi87 filipi87 commented Jan 28, 2026

TTS services improvements.

  • Refactoring TTS services to handle audio context.
  • Adding a parameter to TTSSpeakFrame to conditionally add it to the context.

@codecov
Copy link

codecov bot commented Jan 28, 2026

Codecov Report

❌ Patch coverage is 13.54582% with 217 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/pipecat/services/resembleai/tts.py 0.00% 35 Missing ⚠️
src/pipecat/services/playht/tts.py 0.00% 17 Missing ⚠️
src/pipecat/services/inworld/tts.py 0.00% 14 Missing ⚠️
src/pipecat/services/azure/tts.py 0.00% 13 Missing ⚠️
src/pipecat/services/cartesia/tts.py 0.00% 13 Missing ⚠️
src/pipecat/services/rime/tts.py 0.00% 13 Missing ⚠️
src/pipecat/services/elevenlabs/tts.py 0.00% 12 Missing ⚠️
src/pipecat/services/tts_service.py 60.71% 11 Missing ⚠️
src/pipecat/services/google/tts.py 25.00% 9 Missing ⚠️
src/pipecat/services/asyncai/tts.py 0.00% 8 Missing ⚠️
... and 19 more
Files with missing lines Coverage Δ
src/pipecat/frames/frames.py 88.83% <100.00%> (+0.01%) ⬆️
src/pipecat/processors/aggregators/llm_response.py 73.80% <100.00%> (ø)
...t/processors/aggregators/llm_response_universal.py 77.57% <66.66%> (-0.01%) ⬇️
src/pipecat/services/piper/tts.py 50.00% <66.66%> (ø)
src/pipecat/services/camb/tts.py 0.00% <0.00%> (ø)
src/pipecat/services/fish/tts.py 0.00% <0.00%> (ø)
src/pipecat/services/hathora/tts.py 0.00% <0.00%> (ø)
src/pipecat/services/hume/tts.py 0.00% <0.00%> (ø)
src/pipecat/services/kokoro/tts.py 0.00% <0.00%> (ø)
src/pipecat/services/lmnt/tts.py 0.00% <0.00%> (ø)
... and 21 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@markbackman
Copy link
Contributor

@filipi87 I started work on this here:
#3465

Aleix raised this issue that prompted me to stop work on this:
#3465 (comment)

@filipi87 filipi87 marked this pull request as ready for review January 28, 2026 20:53
@filipi87 filipi87 marked this pull request as draft January 29, 2026 15:25
@filipi87
Copy link
Contributor Author

@markbackman, @aconchillo,

I am halfway through validating that all the TTS services are working after this refactor. I am only missing tests for 12 more TTS services.

However, I believe this PR already addresses the main concern Aleix mentioned in the other PR:
https://github.com/pipecat-ai/pipecat/pull/3465/changes#r2696581006

If you could already provide a review for this one, it would be greatly appreciated.

I will also bring in any other commits that Mark made there which I haven’t addressed in this PR yet.

@filipi87 filipi87 changed the title Added a parameter to TTSSpeakFrame to conditionally add it to the context. TTS services improvements. Jan 30, 2026
@filipi87
Copy link
Contributor Author

@markbackman, @aconchillo, I think it’s now fully ready for review.
I’ll add the changelog later, once we’re happy with all the changes here.

# Converts the text to audio.
@abstractmethod
async def run_tts(self, text: str) -> AsyncGenerator[Frame, None]:
async def run_tts(self, text: str, context_id: str) -> AsyncGenerator[Frame, None]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should context_id be Optional here? Without that, this is a breaking change for run_tts(). While you've covered all cases, there are developers that have built their own services that we should support.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mark, to be honest, I am having second thoughts about making this optional.

I know this would allow TTS services in third party projects to remain compatible with the current signature, but it would also introduce a downside: all the TTS services here would need to add extra logic to handle an optional context_id even though, in practice, it would never be optional.

So I think we should introduce this breaking change now, before 1.0.0. All the TTS services inside Pipecat will already be refactored to handle this anyway.

cc: @aconchillo for thoughts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants