ElevenLabs voice chatbot example with Galileo Protect #126
Conversation
- Terminal-based voice chat using ElevenLabs Conversational AI - Galileo Protect integration for real-time input/output guardrails - Galileo Observe for conversation logging and tracing - Session metrics tracking (latency, turn counts) - Script to create Galileo Protect stage with toxicity rules
There was a problem hiding this comment.
Can you link from the top level README please
| galileo_api_key: str = "" | ||
| galileo_console_url: str = "http://localhost:3000" # Local Galileo instance | ||
| galileo_project_name: str = "elevenlabs-voice-poc" | ||
| galileo_log_stream: str = "voice-conversations" |
There was a problem hiding this comment.
These should load from the env variables so why hard code here?
Also the console doesn't run locally for our users.
|
|
||
| 1. Install system dependencies: | ||
| ```bash | ||
| brew install portaudio # Required for audio support on macOS |
There was a problem hiding this comment.
What about Windows or Linux? Most of our enterprise customers are on Windows, so do they need to install anything.
This should be tagged as macOS only
|
|
||
| ## Requirements | ||
|
|
||
| - Python 3.9+ |
There was a problem hiding this comment.
| - Python 3.9+ | |
| - Python 3.10+ |
We are updating Galileo to 3.10
|
|
||
| ## Learn More | ||
|
|
||
| - [Galileo Documentation](https://docs.galileo.ai/) |
There was a problem hiding this comment.
| - [Galileo Documentation](https://docs.galileo.ai/) | |
| - [Galileo Documentation](https://v2docs.galileo.ai/what-is-galileo) |
| from config import get_settings | ||
|
|
||
| # Set Galileo environment variables BEFORE importing galileo | ||
| settings = get_settings() |
There was a problem hiding this comment.
Seeing as these values are loaded from a .env file, then set as env vars, wouldn't this code be simpler if this was removed?
| print(f"[ERROR] Failed to load configuration: {e}") | ||
| print("\nMake sure you have a .env file with:") | ||
| print(" ELEVENLABS_API_KEY=your-api-key") | ||
| print(" ELEVENLABS_AGENT_ID=your-agent-id") |
There was a problem hiding this comment.
If we are printing what is needed, what about the Galileo API key?
| ws_url=settings.elevenlabs_ws_url, | ||
| ) | ||
|
|
||
| # Set up optional callbacks for Phase 2 Galileo integration |
There was a problem hiding this comment.
If this is a phase 2 thing, then can we remove these?
There was a problem hiding this comment.
There's a lot here and this feels overly complicated, especially how you are manually managing traces and logging spans. Is there a way to make this more concise?
|
|
||
| # Simulate Galileo Protect API call (requires Enterprise for real invoke_protect) | ||
| transcript_preview = transcript[:50] + "..." if len(transcript) > 50 else transcript | ||
| print(f'[GALILEO PROTECT] invoke_protect(stage="voice-guardrails", input="{transcript_preview}")') |
There was a problem hiding this comment.
There's something asynchronous with this protect call - if I do something toxic I get back both a response from eleven labs as well as the protect response. Which feels like the wrong process - the guardrail should stop the text going to the agent to be processed.
There was a problem hiding this comment.
You're correct, this is due to Elevenlabs' event-driven and streaming voice agent architecture, Galileo Protect guardrail checks run asynchronously after user speech is transcribed. As a result, the ElevenLabs agent may begin generating or streaming a response before the guardrail evaluation completes. If a guardrail is triggered, the session is terminated and an override message is played, but a partial or complete agent output may already have been emitted/spoken. We chose this route at the time due to set of requirements by a customer which allowed for this behavior, but I realize that it might not be suitable here. There is a workaround - to be tested/validated - using Elevnlabs' multi-context WebSocket API that can provide more control but would require manual websocket connection management, handling transcription results, running guardrails synchronously and only then send text to their agent context and we would be managing audio contexts for responses. I can try to test/validate that, or we can state this limitation in the README.
It also might be more prudent to close this PR and work on validating the approach using their multi-context WebSocket API and understand trade-offs involved.
|
Closing this PR. Removing all Protect logic and only using Galileo logging and tracing. |
2 participants
ElevenLabs voice chatbot example with Galileo Protect
Before submitting
.env.examplefilePR checklist
Description
Real-time voice chat using Elevenlabs with integration to Galileo observability and Runtime Protect. Sample app uses the terminal to render the conversation. Both user's and chatbot's input and output are evaluated by Galileo. If a toxic and or PII information gets exchanged we disconnect the chat as soon as it is detected.
Type of example
Language
Key features
Testing
Screenshots/Demo
11labs-galileo-protect.mp4