Notes on two-way-audio streamer

[amigniter]

Notes on using Two-Way audio feature (bi-directional streaming) :

Raw binary stream

• When streaming raw binary data over WebSocket, ensure the sample rate matches the module's requirements.
• No additional configuration is needed beyond proper sample rate settings.

JSON base64 encoded stream

The module automatically handles all necessary processing, including buffering, queuing, resampling (up or down), and transcoding to match the streaming channel.

The streaming channel can use any codec supported by FreeSWITCH. During playback, the module will automatically transcode the audio to match the channel's codec. Transcoding time is measured in microseconds, ensuring extremely fast processing for smooth real-time playback. In addition to ultra-fast transcoding, the module further optimizes performance when the returned audio is in G.711 format.

• Since G.711 is the most commonly used codec (the module accepts raw, PCMU, and PCMA), if the returned audio is in PCMU or PCMA format, the module optimizes processing by skipping both resampling and transcoding, playing it back directly.
• You can be sure this optimization will be applied if the channel uses PCMU or PCMA and you return the audio in the same format.
• If returning audio in PCMU or PCMA isn’t possible, the audio should be returned in RAW format.
• If the channel uses a codec other than PCMU or PCMA and you return audio in PCMU/PCMA, there won’t be any crashes or anything like that, but the audio won’t be played back. There’s no real reason to return PCMU/PCMA in such case, so it’s best to avoid it for proper playback and simply return the RAW format instead.

Thank you for using the module! We hope this information helps you optimize your setup for smooth playback. If you have any questions or need further assistance, feel free to reach out.

Happy streaming!

[amigniter]

Tests were done with OpenAI real-time API and ElevenLabs conversational AI API.

Here is a simple websocket server in node.js with ElevenLabs API as an example. You can test the speech-to-speech functionality by downloading v1.0.2 module from releases section (deb package is built for Debian 12) .

Example uses the public agent and you need to insert the agent ID. Also, in this example 8khz (pcm_8000) is used for both the input and output audio, and you need to configure your elevenLabs agent accordingly and modify the websocket server to match the format.

elevenLabs.txt (rename to .js)

[johann-combrink]

would you mind sharing the dial in plan ?

is it just a socket @amigniter ?

<extension name="test">
    <condition field="destination_number" expression="^XXXXXXXXX$">
    <action application="socket" data="localhost:3000 async full"/>
    </condition>
</extension>

[amigniter]

@johann-combrink
Your test dialplan example can be used to pass the control (via ESL) to connector listening on port 3000. From it you could answer the call and exec the API methods.
Dial in plan if used should just answer the call and park it, API calls are handled via the ESL connector.

[andreabat]

Hi there, and congrats for the great work.

I am having problems with the 1.0.02 ( and 1.0.3) module when trying bi-directionality.

I have built a small application ,similar to your elevenlabs example, but using speechmatics as a provider.

Audio is sent via websockets ( to speechmatics ) correctly, and raw responses are returned and written to freeswitch via mod_audio_stream websocket.

But, the returned audio is either muted or is not played on the channel.

I have checked the raw files that are written in the /tmp folder : the resulting audio file seems to be valid ( I used sox to verify).

The freeswitch console log just shows the streamAudio logging, and nothing more ( no codec or playback errors).

I am probably doing something wrong, could you please point me in the right direction ?

I have included a section of the freeswitch log and a resulting wav combined through sox, just to make sure the audio is correct.

audio.zip

fs.log

[os11k]

@andreabat I had same problem today, you probably forgot to set variables, this is how I trigger a call(pay attention to variables set in {}, particularly STREAM_PLAYBACK & STREAM_SAMPLE_RATE )

originate {origination_caller_id_number=+358xxxx,STREAM_PLAYBACK=true,STREAM_SAMPLE_RATE=16000}sofia/internal/371xxx@8.8.8.8 &park

this is how I send media from FS:

uuid_audio_stream <<uuid>> start ws://1.1.1.1:8082/ mono 16k

[eschmidbauer]

is the source code available for streaming playback?

[jfirles]

@amigniter I'm testing the module (1.0.3) with openai realtime api (require pcm 24k) and the response audio from openai is not played but I think that the audio format is ok.

The dialplan:

<extension name="public_extensions"> <condition field="destination_number" expression="^(\d+)$"> <action application="set" data="STREAM_PLAYBACK=true"/> <action application="set" data="STREAM_SAMPLE_RATE=24000"/> <action application="eval" data="${uuid_audio_stream(${uuid} start ws://127.0.0.1:8086 mono 24000)}"/> <action application="park"/> </condition> </extension>

The FreeSWITCH output when the response is sended to the websocket:
2025-05-07 10:24:34.506720 84.37% [DEBUG] audio_streamer_glue.cpp:177 response: {"sampleRate":24000,"audioDataType":"raw","file":"/tmp/fa7a7d85-4abe-4298-8d98-c6e939f582a3_0.tmp.r24"}

If I copy the file, I can play it with this command:

ffplay -f s16le -ar 24k fa7a7d85-4abe-4298-8d98-c6e939f582a3_0.tmp.r24
What am I doing wrong?

I have tested the node example with the same problem.

[dishine]

mod_audio_stream will save the audio file received via WebSocket (WS) to the /tmp/xxx directory and trigger an event notification with Event-Subclass as mod_audio_stream::play. To play the audio file, you need to use ESL (Event Socket Library) with a command like:

api uuid_broadcast uu_id /tmp/xxxx both  

[eschmidbauer]

i believe the streaming response code is not available in this repo. I was able to implement it on my own in a fork. Feel free to reach out directly to me if you need directions on how to implement it.

[amigniter]

@jfirles
STREAM_SAMPLE_RATE is required if you send back raw binary stream. You should remove it if JSON is used.
If you send JSON with b64 audio only STREAM_PLAYBACK is sufficient (with a correct returned JSON format).

[jfirles]

@amigniter my problem was the module version. I thought I was running version 1.0.3, but no, I was running version 1.0.0. Sorry about that.

Now I have put version 1.0.3 but the audio I receive is not in the correct format and OpenAI does not accept it. I have also tried version 1.0.2.
My FreeSWITCH is compiled from source and what I am doing is extracting the ‘mod_audio_stream.so’ file and using it in my installation.

Best regards

[ivonpint]

I’ve successfully integrated mod_audio_stream with a Node.js middleware that relays audio to the Deepgram Agent SDK and plays back responses in real-time using the streamAudio JSON format. Everything is working great — audio streams, responses are heard.

I’m now trying to implement VAD-based interrupt handling, where if the user starts speaking while the agent is still responding, I want to interrupt the agent’s playback and resume capturing the user's voice without waiting for the full response.

From your documentation, I see:

uuid_audio_stream send_text can send metadata to the WebSocket server.

uuid_audio_stream stop can stop and send a final metadata message.

uuid_audio_stream pause and resume exist.

However, I’m unclear how these map to handling real-time interruptions in an active session.

What is the recommended way to signal an interruption to the WebSocket server and ensure the agent playback is canceled or cleared?

If the agent supports it can mod_audio_stream reliably send that metadata using send_text, or is there a better hook for this type of signaling?

Can pause or stop be triggered dynamically from within a Lua script (based on external input or VAD events), or is that expected to be driven externally (e.g., via ESL)?

Is there a way for mod_audio_stream to emit user-speech-started events directly, or should we infer this solely from VAD in the agent layer?

[ivonpint]

@jfirles
Here's a simple dialplan and lua script to get it working make sure to build 1.02 or install the deb package from the repo

----------------------------- LUA ---------------------------------------

package.path = package.path .. ";/usr/share/freeswitch/scripts/?.lua"
local api = freeswitch.API()

session:setVariable("STREAM_PLAYBACK", "true")
session:setVariable("STREAM_SAMPLE_RATE", "24000")

session:answer()

local uuid = session:get_uuid()
local cmd = string.format("uuid_audio_stream %s start ws://127.0.0.1:8086 mono 24000", uuid)
api:executeString(cmd)

session:execute("park", "")

---

DialPlan.xml:

 <extension name="AI_Assistant">
    <condition field="destination_number" expression="^(\d+)$">
      <action application="answer"/>
      <action application="lua" data="ai.lua"/>
    </condition>
  </extension>

This will work as long as your AI Agent API is sending you data like this:

jsonPayload = {
      type: 'streamAudio',
      data: {
        audioDataType: 'raw',
        sampleRate: 16000,
        audioData: base64Audio
      }

[Hamza962]

Hi @amigniter,

I’ve just installed version 1.0.3 and successfully connected to OpenAI’s Realtime API. I'm able to send and receive audio through the conversational API, but the audio is not being played back to the caller.

However, if I play back temporary raw audio files using the uuid_broadcast API method via ESL, the audio plays correctly.

Could you please review the details below and advise on what might be going wrong?

DIALPLAN :

<extension name="audio_stream"> <condition field="destination_number" expression="^8888$"> <action application="set" data="STREAM_PLAYBACK=true"/> <action application="set" data="call_tag=audio_stream"/> <action application="answer"/> <action application="export" data="execute_on_answer=lua esl_stream.lua"/> <action application="park"/> </condition> </extension>

FS LOGS :

2025-05-11 12:06:01.560968 99.43% [DEBUG] mod_audio_stream.c:81 calling stream_session_init.
2025-05-11 12:06:01.560968 99.43% [DEBUG] audio_streamer_glue.cpp:386 (e3090bca-a001-43d0-b880-5a69e3c2031c) resampling from 8000 to 16000
2025-05-11 12:06:01.560968 99.43% [DEBUG] audio_streamer_glue.cpp:397 (e3090bca-a001-43d0-b880-5a69e3c2031c) stream_data_init
2025-05-11 12:06:01.560968 99.43% [DEBUG] mod_audio_stream.c:87 adding bug.
2025-05-11 12:06:01.560968 99.43% [DEBUG] switch_core_media_bug.c:976 Attaching BUG to sofia/internal-5161/100@demo.pbx.com
2025-05-11 12:06:01.560968 99.43% [DEBUG] mod_audio_stream.c:91 setting bug private data.
2025-05-11 12:06:01.560968 99.43% [DEBUG] mod_audio_stream.c:94 exiting start_capture.
2025-05-11 12:06:01.560968 99.43% [DEBUG] audio_streamer_glue.cpp:163 sending initial metadata "e3090bca-a001-43d0-b880-5a69e3c2031c"
2025-05-11 12:06:05.820942 99.33% [DEBUG] audio_streamer_glue.cpp:195 response: {"audioDataType":"raw","sampleRate":8000,"file":"/tmp/e3090bca-a001-43d0-b880-5a69e3c2031c_0.tmp.r8"}
2025-05-11 12:06:05.860949 99.33% [DEBUG] audio_streamer_glue.cpp:195 response: {"audioDataType":"raw","sampleRate":8000,"file":"/tmp/e3090bca-a001-43d0-b880-5a69e3c2031c_1.tmp.r8"}
2025-05-11 12:06:06.020947 99.33% [DEBUG] audio_streamer_glue.cpp:195 response: {"audioDataType":"raw","sampleRate":8000,"file":"/tmp/e3090bca-a001-43d0-b880-5a69e3c2031c_2.tmp.r8"}
2025-05-11 12:06:06.040950 99.33% [DEBUG] audio_streamer_glue.cpp:195 response: {"audioDataType":"raw","sampleRate":8000,"file":"/tmp/e3090bca-a001-43d0-b880-5a69e3c2031c_3.tmp.r8"}
2025-05-11 12:06:06.120941 99.30% [DEBUG] audio_streamer_glue.cpp:195 response: {"audioDataType":"raw","sampleRate":8000,"file":"/tmp/e3090bca-a001-43d0-b880-5a69e3c2031c_4.tmp.r8"}
2025-05-11 12:06:06.180946 99.30% [DEBUG] audio_streamer_glue.cpp:195 response: {"audioDataType":"raw","sampleRate":8000,"file":"/tmp/e3090bca-a001-43d0-b880-5a69e3c2031c_5.tmp.r8"}
2025-05-11 12:06:06.240941 99.30% [DEBUG] audio_streamer_glue.cpp:195 response: {"audioDataType":"raw","sampleRate":8000,"file":"/tmp/e3090bca-a001-43d0-b880-5a69e3c2031c_6.tmp.r8"}
2025-05-11 12:06:06.460961 99.30% [DEBUG] audio_streamer_glue.cpp:195 response: {"audioDataType":"raw","sampleRate":8000,"file":"/tmp/e3090bca-a001-43d0-b880-5a69e3c2031c_7.tmp.r8"}
2025-05-11 12:06:06.540961 99.30% [DEBUG] audio_streamer_glue.cpp:195 response: {"audioDataType":"raw","sampleRate":8000,"file":"/tmp/e3090bca-a001-43d0-b880-5a69e3c2031c_8.tmp.r8"}
2025-05-11 12:06:10.600953 99.37% [NOTICE] sofia.c:1065 Hangup sofia/internal-5161/100@demo.pbx.com [CS_EXECUTE] [NORMAL_CLEARING]

PYTHON FUNCTION TO SEND RAW AUDIO BACK TO MODULE :

async def forward_audio_to_client(openai_ws, client_ws):
    async for msg in openai_ws:
        if msg.type == aiohttp.WSMsgType.TEXT:
            data = json.loads(msg.data)
            if data.get("type") == "response.audio.delta":
                logging.info(f"[OPENAI] Received audio delta : {data}")
                try:
             
                    audio_b64 = data["delta"]
                    raw_audio = base64.b64decode(audio_b64)

                  
                    audio_segment = AudioSegment(
                        data=raw_audio,
                        sample_width=2,  # 16-bit PCM
                        frame_rate=24000,
                        channels=1
                    )

                    # Resample to 8kHz
                    audio_segment_8k = audio_segment.set_frame_rate(8000)

                    # Export as RAW format with 8kHz sample rate
                    raw_io = BytesIO()
                    audio_segment_8k.export(raw_io, format="raw")
                    raw_data = raw_io.getvalue()

                   
                    raw_b64 = base64.b64encode(raw_data).decode("utf-8")

           
                    response_msg = {
                        "type": "streamAudio",
                        "data": {
                            "audioDataType": "raw", 
                            "sampleRate": 8000,
                            "audioData": raw_b64
                        }
                    }
                    await client_ws.send(json.dumps(response_msg))
                    logging.info("[OPENAI] Sent raw 8kHz audio to client")
                except Exception as e:
                    logging.error(f"[ERROR] Audio conversion failed: {e}")