[Bug] v7.0.128 stuck in 100% CPU and no data being sent in SRT publishing

[felipewd]

Hi,

I'm using v7.0.128 to receive 2 SRT streams. The ffmpeg processing I'm doing outside srs.

However during the night those streams reconnected and srs entered in a bad place: spinning 100% CPU, stuck, and wouldn't allow me to publish any new streams or ffprobe the current ones. I had to kill it and restart it get it going.

I was able to capture perf data here srs.txt

It wad gathered with:

perf record -F 199 -g -p $(pidof srs) -- sleep 30
perf script > srs.txt

The offending stack seems to be:

srt::CUDT::tsbpd(void*)
srt::sync::Condition::wait_until()
srt::sync::steady_clock::now()
pthread_mutex_unlock
clock_gettime

And also got a perf top:

Now, all things point to a TSBPD issue...which is srt-related.

The main problem for me was srs not being able to publish any other SRT streams, wouldn't drop the troubling one (for whatever reason), and I couldn't ffprobe any streams.

There weren't any much useful log lines, however I noticed the last one were from last night, so it seems even the log file wasn't being updated anymore:

[2025-11-18 23:03:40.220][INFO][784764][87185zl5] -> SRT_PLA Transport Stats # pktSent=48434, pktSndLoss=0, pktRetrans=0, pktSndDrop=0
[2025-11-18 23:03:40.220][INFO][784764][87185zl5] -> SRT_PLA time=3656994, packets=48434, okbps=0,10199,10199, ikbps=0,0,0
[2025-11-18 23:03:42.648][INFO][784764][057080l6] SRS: cpu=13.99%,48MB, cid=2,0, timer=56,0,0, clock=1,27,7,3,2,1,1,0,0, objs=(pkt:0,raw:0,fua:0,msg:5810,oth:0,buf:0)
[2025-11-18 23:03:47.650][INFO][784764][057080l6] SRS: cpu=13.99%,48MB, cid=2,0, timer=62,0,0, clock=0,42,6,0,0,0,0,0,0, objs=(pkt:0,raw:0,fua:0,msg:5811,oth:0,buf:0)
[2025-11-18 23:03:48.193][INFO][784764][32m36124] <- SRT_CPB Transport Stats # pktRecv=9681, pktRcvLoss=0, pktRcvRetrans=0, pktRcvDrop=0
[2025-11-18 23:03:48.193][INFO][784764][32m36124] <- SRT_CPB time=3671853, packets=9680, okbps=0,0,0, ikbps=0,10193,10199
[2025-11-18 23:03:50.236][INFO][784764][585vt76u] -> SRT_PLA Transport Stats # pktSent=29069, pktSndLoss=0, pktRetrans=0, pktSndDrop=0
[2025-11-18 23:03:50.236][INFO][784764][585vt76u] -> SRT_PLA time=3660920, packets=29069, okbps=0,10199,10200, ikbps=0,0,0
[2025-11-18 23:03:52.654][INFO][784764][057080l6] SRS: cpu=14.99%,48MB, cid=2,0, timer=62,0,0, clock=0,42,6,0,0,0,0,0,0, objs=(pkt:0,raw:0,fua:0,msg:5811,oth:0,buf:0)
[2025-11-18 23:03:57.657][INFO][784764][057080l6] SRS: cpu=13.99%,48MB, cid=2,0, timer=62,0,0, clock=0,40,7,0,0,0,0,0,0, objs=(pkt:0,raw:0,fua:0,msg:5815,oth:0,buf:0)
[2025-11-18 23:03:58.207][INFO][784764][6534sfev] <- SRT_CPB Transport Stats # pktRecv=38775, pktRcvLoss=0, pktRcvRetrans=0, pktRcvDrop=0
[2025-11-18 23:03:58.207][INFO][784764][6534sfev] <- SRT_CPB time=3681861, packets=38768, okbps=0,0,0, ikbps=0,10199,10200
[2025-11-18 23:04:00.659][INFO][784764][057080l6] CircuitBreaker: cpu=200.60%,48MB, break=1,1,0, cond=200.60%
[2025-11-18 23:04:01.659][INFO][784764][057080l6] CircuitBreaker: cpu=201.00%,48MB, break=1,1,0, cond=201.00%
[2025-11-18 23:04:02.659][INFO][784764][057080l6] CircuitBreaker: cpu=200.00%,48MB, break=1,1,0, cond=200.00%
[2025-11-18 23:04:02.659][INFO][784764][057080l6] SRS: cpu=200.00%,48MB, cid=2,0, timer=62,0,0, clock=0,40,7,0,0,0,0,0,0, objs=(pkt:0,raw:0,fua:0,msg:5815,oth:0,buf:0)
[2025-11-18 23:04:03.660][INFO][784764][057080l6] CircuitBreaker: cpu=200.80%,48MB, break=1,1,0, cond=200.80%

here's my full config:

listen              1955;
max_connections     1000;
srs_log_tank        file;
srs_log_file        ./log/srs.log;
pid		    log/srs.pid;
daemon              on;
http_api {
    enabled         on;
    listen          1985;
}

stats {
    network         0;
}

srt_server {
    enabled          on;
    listen           1935;
    maxbw           -1;
    mss              1456;
    latency          800;
    recvlatency      800;
    peerlatency      800;
    tlpktdrop        off;
    sendbuf          8388608;
    recvbuf          8388608;
    peer_idle_timeout 15000;
    connect_timeout    8000;
    tsbpdmode       on;
    default_app     live;
}


vhost __defaultVhost__ {
     min_latency off;
     tcp_nodelay off;
     chunk_size 128;
     in_ack_size 0;
     out_ack_size 2500000;
     publish {
	     mr off;
	     mr_latency 350;
	     firstpkt_timeout 20000;
	     normal_timeout 7000;
	     parse_sps on;
	     try_annexb_first on;
	     kickoff_for_idle 0;
     }
     play {
	     gop_cache off;
	     gop_cache_max_frames 2500;
	     queue_length 10;
	     time_jitter full;
	     atc off;
	     mix_correct off;
	     atc_auto off;
	     mw_latency 350;
	     mw_msgs 8;
	     send_min_interval 10.0;
	     reduce_sequence_header on;
     }
     srt {
	     enabled on;
	     srt_to_rtmp off;
     }
}

I can provide my current 2 streams if you want...but I don't know what packet/network scenario cause things to haywire on srs. So I can't reproduce the issue.

[winlinvip]

If the issue is from the libsrt library, you should file an issue there, because we never touch the libsrt code in SRS.

[felipewd]

I'm not sure it is, please enlighten me :-)

I just wanted to point out that srs being completely stuck on a bogus stream (or network scenario) is not ideal, right? Even the log output got stuck, apparently.

But please, let me know what can I do to help track this down.

[felipewd]

I'm building it against the repo 3rdparty/srt-1-fit. Should I update it there? If so, how?

[felipewd]

I see this commit ab1acaf from @duiniuluantanqin updating the lib to 1.5.3. The repo from Haivision has a 1.5.4 release since Nov 11, 2024 and 51 commits there since (https://github.com/Haivision/srt/releases/tag/v1.5.4). Maybe the lib in the repo should be updated? I can test if you guys want, just let me know how.

But if it's a libsrt for sure, then by all means...I'd like to help fix it so srs doesn't pay the price and get stuck like this.

[winlinvip]

Could you please try to upgrade to 1.5.4 and check if it fix this issue or not?

[felipewd]

sure thing, I rebuild it against the new srt repo....and will leave it for a few days (it took around 5-7 days for the issue to appear, so I'm not sure how long it would take)

[artem-smorodin-dacast]

I encountered the same problem today, but only after using Streamlabs for an hour.

SRS seems to be completely stucked, even the API is not responding.

My srs version - 6.0.183

[artem-smorodin-dacast]

@winlinvip It seems that even updating libsrt to the latest master does not fix this.

I haven't found anything related in the libsrt issue, so I assume it's related to the integration of libsrt into srs.

[winlinvip]

Nop, it's also possible be caused by a bug still exists in libsrt 1.5.4.

[artem-smorodin-dacast]

@felipewd @winlinvip I may have found the cause of this bug. It may be caused by the combination of tsbpdmode on + tlpktdrop off. SRT server receives late packets but does not discard them, and after a while tsbpd buffer overflows, causing a livelock. I mean that there are so many packets that iterating through this buffer takes up 100% of the CPU time.

I still need some time to test stability, but if I used to get a stuck after about 2-3 hours, after enabling tlpktdrop I've been fine for about 12 hours.

I don't know if enabling tlpktdrop fixes this bug completely or just gives me more time, I'll get back to you when I test it.

[Adam-Burke]

I have a similar issue with the below config. I tried 1.5.5 rc0 and it didn't seem to help

srt_server {
    enabled on;
    listen 10080;
    connect_timeout 6000;
    recvlatency 1000;
    tlpktdrop off;
    tsbpdmode on;
    mix_correct      on;
    peer_idle_timeout 2000;
}

[artem-smorodin-dacast]

I can confirm that I no longer experience any issues after enabling tlpktdrop, even without updating libsrt version.

I think we should update the documentation and mention this known issue.

[winlinvip]

What's the root cause? No need to desperately update the document unless we know the root cause.