Load test with sipp

[Pandi1981]

While sending load test from sipp we are facing socket connection reestablish every time in sipp side but in the callee side we are playing some music and no issue with the rtp stream. But calller is sipp facing re trying socket connection every time... Please suggest me your view....

No issue when we are making call from zoiper Or any other sip clients

[Pandi1981]

When we enable RTP stream through LIA script freeswitch getting crashee with out RTP stream freeswitch working upto 400 concurrent call... With rtp stream with 50 concurrent call freeswitch crahsed with high cpu... Any suggestion please

[Pandi1981]

Issue resolved!!! But we could not able to achieve not more that 70 concirrent call in the freeswitch 8 core 16 gb ram getting 700% cpu and freeswitch got restart... Any suggestions please.

[Pandi1981]

stream_audio.zip
mod_cpp.zip
Hi team,
I have modified the source code for my requirements(start stop notifications) and the streaming i am doing using lua script.
During load test we have observed CPU utilization is very high during initial. when RTP is established CPU utilization comes very normal.
Please let me know if i am doing anything wrong in the attached files.
Please help me little bit urgent!!!!

[Pandi1981]

While running valgrind I am getting the below message
==16102== Thread 32:
==16102== Conditional jump or move depends on uninitialised value(s)
==16102== at 0xA0D732F: AudioStreamer::AudioStreamer(char const*, char const*, void ()(switch_core_session, char const*, char const*), int, int, char const*, bool, bool, char const*, char const*, char const*, char const*) (in /usr/local/freeswitch/mod/mod_audio_stream.so)
==16102== by 0xA0D5B12: stream_session_init (in /usr/local/freeswitch/mod/mod_audio_stream.so)
==16102== by 0xA0D3CEE: stream_function (in /usr/local/freeswitch/mod/mod_audio_stream.so)
==16102== by 0x4948296: switch_api_execute (switch_loadable_module.c:2968)
==16102== by 0x49E3D25: API::execute(char const*, char const*) (switch_cpp.cpp:248)
==16102== by 0xA7F7E10: ??? (mod_lua_wrap.cpp:3577)
==16102== by 0xA81D118: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA828AB6: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA81D7A4: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA81CA9A: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA81D94F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA818D77: lua_pcall (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102==

=16102== Thread 34 wss://loadtest-:
==16102== Invalid read of size 1
==16102== at 0xA0D9236: std::_Function_handler<void (std::unique_ptr<ix::WebSocketMessage, std::default_deleteix::WebSocketMessage > const&), AudioStreamer::AudioStreamer(char const*, char const*, void ()(switch_core_session, char const*, char const*), int, int, char const*, bool, bool, char const*, char const*, char const*, char const*)::{lambda(std::unique_ptr<ix::WebSocketMessage, std::default_deleteix::WebSocketMessage > const&)#1}>::_M_invoke(std::_Any_data const&, std::unique_ptr<ix::WebSocketMessage, std::default_deleteix::WebSocketMessage > const&) (in /usr/local/freeswitch/mod/mod_audio_stream.so)
==16102== by 0xA0E08AB: ix::WebSocket::connect(int) (in /usr/local/freeswitch/mod/mod_audio_stream.so)
==16102== by 0xA0E0F1D: ix::WebSocket::checkConnection(bool) (in /usr/local/freeswitch/mod/mod_audio_stream.so)
==16102== by 0xA0E1DA7: ix::WebSocket::run() (in /usr/local/freeswitch/mod/mod_audio_stream.so)
==16102== by 0x5AE24A2: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==16102== by 0x4D93133: start_thread (pthread_create.c:442)
==16102== by 0x4E12A3F: clone (clone.S:100)
==16102== Address 0x7c65048 is 168 bytes inside a block of size 232 free'd
==16102== at 0x484317B: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==16102== by 0xA0D350A: stream_function (in /usr/local/freeswitch/mod/mod_audio_stream.so)
==16102== by 0x4948296: switch_api_execute (switch_loadable_module.c:2968)
==16102== by 0x49E3D25: API::execute(char const*, char const*) (switch_cpp.cpp:248)
==16102== by 0xA7F7E10: ??? (mod_lua_wrap.cpp:3577)
==16102== by 0xA81D118: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA828AB6: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA81D7A4: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA81CA9A: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA81D94F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA818D77: lua_pcall (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA7DC3D1: docall (mod_lua.cpp:92)
==16102== Block was alloc'd at
==16102== at 0x48407B4: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==16102== by 0x4DA88E9: strdup (strdup.c:42)
==16102== by 0xA0D3397: stream_function (in /usr/local/freeswitch/mod/mod_audio_stream.so)
==16102== by 0x4948296: switch_api_execute (switch_loadable_module.c:2968)
==16102== by 0x49E3D25: API::execute(char const*, char const*) (switch_cpp.cpp:248)
==16102== by 0xA7F7E10: ??? (mod_lua_wrap.cpp:3577)
==16102== by 0xA81D118: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA828AB6: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA81D7A4: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA81CA9A: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA81D94F: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102== by 0xA818D77: lua_pcall (in /usr/lib/x86_64-linux-gnu/liblua5.1.so.0.0.0)
==16102==

[Pandi1981]

Please give me some input please!!!! Please find the dial plan...

<extension name="PSTN-outbound-logimeter-landto-IVR">
  <condition field="destination_number" expression="^711$">
    <action application="set" data="STREAM_B64_AUDIO=true"/>
    <action application="set" data="STREAM_BUFFER_SIZE=20"/>
    <action application="set" data="my_uuid=${uuid}" inline="true"/>
    <action application="set" data="sip_copy_custom_headers=true"/>
    <action application="set" data="sip_h_X-outbound-callid=${uuid}"/>
    <action application="set" data="export_vars=${sip_call_id}"/>
    <action application="set" data="customerno=${destination_number}" inline="true"/>
    <action application="set" data="sip_h_X-calltype=${sip_h_X-calltype}"/>

    <action application="set" data="api_on_answer=luarun stream_audio.lua ${my_uuid} ${sip_h_X-calltype} ${destination_number} ${sip_from_user} ${sip_h_X-agent_id} ${sip_h_X-callid}"/>

    <action application="bridge" data="sofia/gateway/opensips/${destination_number}"/>
   </condition>
</extension>

[Pandi1981]

Any one tried load test with mod_audio_stream

[mrunknown-art]

@Pandi1981 I am using the version 1.0.2 the same way you are using and doing 2 way audio using json event over the websocket. I am facing memory issue . After call hangup and socket disconnect freeswitch not releasing the memory and call per call it's piling up to 100%. after restarting freeswitch it's coming normal. Please guide .

[undev306]

Hello, also using version 1.0.2 and getting memory peak simply putting a test scenario of 20 concurrent calls each 10 minutes with fewer than 1 minute of stream each

FS memory with 2 reboots happening recently:

Highly suspecting the module not releasing memory over time. important question, if somehow my call scenario fails and the user doesn't stop the stream, will the stream automatically stop itself on call hangup?

[amigniter]

@undev306

Highly suspecting the module not releasing memory over time. important question, if somehow my call scenario fails and the user doesn't stop the stream, will the stream automatically stop itself on call hangup?

Definitely the memory is managed properly, both in the module and the websocket client.
Yes, the stream will be stopped on hangup.

Which version are you actually using, custom built module from the github, or downloaded commercial version?
Please use v1.0.3 if you are using the commercial edition, the latest version.

[undev306]

@undev306

Highly suspecting the module not releasing memory over time. important question, if somehow my call scenario fails and the user doesn't stop the stream, will the stream automatically stop itself on call hangup?

Definitely the memory is managed properly, both in the module and the websocket client. Yes, the stream will be stopped on hangup.

Which version are you actually using, custom built module from the github, or downloaded commercial version? Please use v1.0.3 if you are using the commercial edition, the latest version.

sorry I missed the context: using the v1.0.2 custom built module from github (the exemple shows a built of a few weeks ago with FS 1.10.11.

Nice to know it stops the stream on hangup 🆒
Any advice on how to debug properly as this module does not have its own pid to track memory on ?
Thanks

[amigniter]

@undev306

Thank you for your post.
I was kind of curious by your first post and the memory usage (gigabytes used there) so I had to run some SIPp tests myself to actually see how it goes. Totally different story here. My FreeSWITCH is pretty much stripped down, only a few essential modules loaded so it initially uses much less RAM.

Any advice on how to debug properly as this module does not have its own pid to track memory on ?

No, you can't just examine the module but the FreeSWITCH itself (by its PID) and the memory usage.

in this SIPp test 15cps used with 50 concurrent calls; Call duration 3 - 8s with total of 500 calls. As you can see the memory peak is at ~83 - 85 MB. (sample line is in seconds)

In 2nd test increased to 50cps and 100 concurrent calls, same duration, total 1000 calls. Initial RSS: ~83 MB, Peak RSS: ~145 MB with increase of ~63 MB , about 0.6 MB per call (630 KB/call).
In both tests played PCMU (G.711 μ-law)

FreeSWITCH v1.10.12 used and the mod_audio_stream v1.0.3. If you are interested I can upload tomorrow the SIPp scenario and everything used for benchmarking.

[undev306]

Hi thanks @amigniter, I built FS from source by the way. Would like to get the hands on this scenario sure :) and will give you a feedback

[amigniter]

A bit heavier test done now. 40 calls per second aiming for 1000 concurrent calls, actual peak was 731 parallel calls. Call duration 5-30s with PCMU audio transmitted.

This graph is phenomenal from a stability and leak-detection standpoint. It shows exactly what a well-behaved media system should look like under a heavy sustained load.
From sample ~80 to ~280, RSS stabilizes between 500–520 MB and stays flat. Measured CPU usage was in range between 19-22% (i5, 6-cores, 12-threads).

Scenario used, audio_stream_scenario.xml

Dialplan (used in public context)

<extension name="sipp-stream-test">
  <condition field="destination_number" expression="^5000$">
  <action application="set" data="STREAM_PLAYBACK=false"/> 
   <action application="eval" data="${uuid_audio_stream(${uuid} start ws://192.168.0.23:3001 mono 8k)}"/>
   <action application="answer"/>
    <action application="park"/>
  </condition>
</extension>

[amigniter]

To create a graph, an approach used (linux / (k)ubuntu):

Start pidstat with 1s interval before starting SIPp:

pidstat -r -p (FreeSWITCH PID) 1 > mem.log

get it with: ps aux | grep freeswitch

Parse mem.log

grep -v '^Linux' mem.log | grep -v '^$' | grep -v 'UID' | awk '{print $8}' > rss.log

Create plot.gnu file with the content:

set terminal png size 1280,720
set output "mem.png"
set title "FreeSWITCH RSS Memory"
set xlabel "Sample"
set ylabel "RSS (KB)"
plot "rss.log" using 1 with lines title "RSS KB"

Create a graph with gnuplot (needs to be installed first):

gnuplot plot.gnu

After starting pidstat you should run your SIPp tests, an example used:

sudo sipp -sf audio_stream_scenario.xml 192.168.0.19:5060 \
-i 192.168.0.19 \
-p 5062 \
-l 1000 \
-r 40 \
-m 10000

create PCMU pcap for playing with SIPp

At this point it looks even to me it's too complicated, but it is what it is, nothing simple with SIPp. Once you get everything it will be much, much easier to do the benchmarking.

We need to dump some packets, so make a call and inside fs_cli it will log info about the call, and mapped port (UDP)

2025-11-27 11:34:38.981491 96.83% [DEBUG] sofia.c:7493 Channel sofia/internal/1000@192.168.0.19:5061 entering state [ready][200]
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_media.c:5526 Audio Codec Compare [PCMU:0:8000:20:64000:1]/[opus:116:48000:20:0:1]
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_media.c:5526 Audio Codec Compare [PCMU:0:8000:20:64000:1]/[PCMU:0:8000:20:64000:1]
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_media.c:5588 Audio Codec Compare [PCMU:0:8000:20:64000:1] ++++ is saved as a match
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_media.c:5526 Audio Codec Compare [PCMU:0:8000:20:64000:1]/[PCMA:8:8000:20:64000:1]
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_media.c:5438 Set telephone-event payload to 101@8000
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_media.c:3731 Set Codec sofia/internal/1000@192.168.0.19:5061 PCMU/8000 20 ms 160 samples 64000 bits 1 channels
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_codec.c:111 sofia/internal/1000@192.168.0.19:5061 Original read codec set to PCMU:0
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_media.c:5798 Set telephone-event payload to 101@8000
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_media.c:5856 sofia/internal/1000@192.168.0.19:5061 Set 2833 dtmf send payload to 101 recv payload to 101
2025-11-27 11:34:38.981491 96.83% [DEBUG] switch_core_media.c:8660 AUDIO RTP [sofia/internal/1000@192.168.0.19:5061] 192.168.0.19 port 27234 -> 192.168.0.19 port 8000 codec: 0 ms: 20

The port is 27234 :

sudo tcpdump -nq -s 0 -i lo -w ./dump.pcap port 27234

FreeSWITCH and the softphone client are on the same machine, so the interface used was lo, make sure to use the correct network interface.

dump.pcap will be your custom pcap file to stream from SIPp. Next steps are not required but if you want to test it out (and actually play it to make sure it is correctl) you can extract raw pcmu packets from dumped pcap:

tshark -r dump.pcap \ -d udp.port==27234,rtp \ -Y "rtp && rtp.p_type==0" \ -T fields -e rtp.payload \ | tr -d ':\n' \ | xxd -r -p > audio.ulaw

and you can play it with aplay:

aplay -f MU_LAW -r 8000 -c 1 audio.ulaw

[undev306]

@amigniter thanks for detailed response, really appreciate your help. Will try the tool as soon as I can