Error during integration agent with eleven labs

[vaskevych]

Try to setup multi languages
{ "type": "Settings", "audio": { "input": { "encoding": "mulaw", "sample_rate": 8000 }, "output": { "encoding": "mulaw", "sample_rate": 8000, "container": "none" } }, "agent": { "language": "multi", "listen": { "provider": { "type": "deepgram", "model": "nova-3" } }, "think": { "provider": { "type": "open_ai", "model": "gpt-4o-mini" }, "prompt": some info, "functions": [ ..... ] }, "speak": { "provider": { "type": "eleven_labs", "model_id": "eleven_multilingual_v2", "language_code": "en" }, "endpoint": { "url": "wss://api.elevenlabs.io/v1/text-to-speech/dcnUh5SzrOkP1aCO1gEr/stream-input", "headers": { "xi-api-key": "xxx863d3" } } } } }
after success connection receive message like this
We did not receive audio within our timeout. Please make sure you are sending binary messages containing user speech, but when use deepgram as speak provider everything is Ok.
I am using C#

[deepgram-community[bot]]

Thanks for asking your question. Please be sure to reply with as much detail as possible so the community can assist you efficiently.
_{Consider joining our Discord community for more opportunity to engage with your fellow Deepgram users. You can earn points which can be redeemed for cool stuff by being active in our communities!}

[deepgram-community[bot]]

It looks like we're missing some important information to help debug your issue. Would you mind providing us with the following details in a reply?

• The deepgram product you are using (e.g Speech to Text, Agent API)
• A request ID that triggered your error or issue.

[vaskevych]

Agent API

[vaskevych]

my request Id: 4bc123bd-6a53-4738-be02-4e40fb532374

[vaskevych]

Any ideas?

[deepgram-community[bot]]

It seems like there might be an issue with how the audio data is being sent to the Eleven Labs API. Since you're not encountering issues when using Deepgram as the speech provider, the issue might be specific to the configuration or handling with Eleven Labs. Here are a few suggestions and troubleshooting steps:

1. Check Audio Encoding and Sample Rate:

   • Ensure that the audio encoding and sample rate match the capabilities and requirements of the Eleven Labs API.

2. WebSocket Connection:

   • Verify that the WebSocket connection to Eleven Labs is established correctly. Check for any connection errors or warnings in the logs.

3. Binary Message Formatting:

   • Confirm that you are sending audio data in the correct format expected by Eleven Labs.
   • Ensure that the audio messages are being sent as binary frames over the WebSocket.
   • Double-check the framing and structure of messages being sent to ensure they comply with the API's expected message protocol.

4. Timeout Settings:

   • Investigate if there's any timeout setting on the client side that might be affecting the audio transmission.
   • Some WebSocket libraries allow you to configure timeouts which might need adjusting.

5. API Key and Endpoint:

   • Ensure that the API key and endpoint configuration are correct and have the right permissions.

6. Implementation in C#:

   • Verify the logic in your C# code to ensure data is being read, processed, and sent correctly.
   • Use logging to trace the data flow and identify where the issue could be occurring.

If these steps do not resolve the issue, it might be helpful to consult the documentation provided by Eleven Labs or reach out to their support for more detailed assistance. Additionally, reviewing specific API usage examples or community forums related to Eleven Labs might provide more context or solutions.

This message was sent by John Vajda from Deepgram, via our community automation.