Changes to support Gemini Live in Pipecat Flows#3620
Open
Conversation
kompfner
force-pushed
the
pk/flows-gemini-live-support
branch
2 times, most recently
from
47940fd to
d39a670
Compare
- Detect when a newly-received context warrants a reconnection to the Gemini Live API; we need to reconnect in order to re-seed new conversation history or swap out the current set of tools. This reconnection occurs when Pipecat Flows transitions between conversational nodes, as context has been edited and/or tools added/removed. - Strip function call and response messages out of context before sending to Gemini Live when seeding conversation history, to sidestep a seeming Gemini Live limitation (see https://stackoverflow.com/a/79851394)
kompfner
force-pushed
the
pk/flows-gemini-live-support
branch
from
d39a670 to
c1ac1a6
Compare
kompfner
requested review from
aconchillo,
filipi87,
markbackman and
mattieruth
…hange to the context is newly-appended messages, send them to the server. This requires us to distinguish between newly-appended "bookkeeping" messages that just reflect what Gemini Live already said, and messages that were programmatically inserted, such as from the the transition to the new Pipecat Flows node. This change makes it so that using `LLMMessagesAppendFrame` will have the desired effect, of updating the Gemini Live conversation.
kompfner
force-pushed
the
pk/flows-gemini-live-support
branch
from
be6612e to
44b917c
Compare
…ing a model response twice when there is a Flows node transition that returns a value from the transition function and loads new context messages in the new node
… triggering a model response twice when there is a Flows node transition that returns a value from the transition function and loads new context messages in the new node" This reverts commit 90e6f0d.
…he case of programmatically-appended context messages, do a full reconnection—I've tried many things to get "live updates" to context working reliably and have been unable to (Gemini code comments also warn against doing "live updates" after the initial context seeding prior to starting audio input).
…trip function call and response messages out of context before sending to Gemini Live when seeding conversation history, which we were doing to sidestep a seeming Gemini Live limitation (see https://stackoverflow.com/a/79851394), convert them to regular text messages with special formatting
… with `LLMSetToolsFrame`
kompfner
commented
Feb 5, 2026
changelog/3620.added.md
Outdated
| @@ -0,0 +1 @@ | |||
| - Added support to Gemini Live for programmatically swapping tools or editing context at runtime; now you can use `LLMMessagesAppendFrame`, `LLMMessagesUpdateFrame`, and `LLMSetToolsFrame` with Gemini Live, like you would with text-to-text services. Importantly, this change enables you to use Gemini Live with Pipecat Flows, if you're using Pipecat Flows 0.0.23 or above. | |||
Contributor
Author
There was a problem hiding this comment.
TODO: update the Pipecat Flows version mentioned here if it turns out the necessary Flows changes don't make it in that version.
kompfner
force-pushed
the
pk/flows-gemini-live-support
branch
from
2d6fcd2 to
c49ee86
Compare
kompfner
force-pushed
the
pk/flows-gemini-live-support
branch
from
c49ee86 to
e039abd
Compare
…ing context in a frame-based way. The previous approach required the caller to directly grab a reference to the context object, grab a "snapshot" of its messages *at that point in time*, transform the messages, and then push an `LLMMessagesUpdateFrame` with the transformed messages. This approach can lead to problems: what if there had already been a change to the context queued in the pipeline? The transformed messages would simply overwrite it without consideration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please read commit messages for change descriptions.
Pipecat Flows companion PR: pipecat-ai/pipecat-flows#232