High CPU usage caused by epoll_wait, how to fix it? #308
Description
CentOS Linux release 7.9.2009 (Core)
1. Executing fs_cli -x 'status'
UP 0 years, 39 days, 18 hours, 30 minutes, 24 seconds, 941 milliseconds, 264 microseconds
FreeSWITCH (Version 1.10.12-dev 64bit) is ready
22204894 session(s) since startup
236 session(s) - peak 1100, last 5min 239
6 session(s) per Sec out of max 600, peak 196, last 5min 24
1200 session(s) max
min idle cpu 20.00/86.13
Current Stack Size/Max 240K/8192K
User Comment: The request rate per second is very low.
- Executing top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9176 root -2 -10 9812.9m 3.6g 7328 S 266.7 23.4 44723:09 freeswitch
User Comment: High CPU usage is observed.
- Then, executing the top -H -p 9176 command, the result is as follows:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9187 root rt -10 9823.6m 3.6g 7328 R 70.9 23.4 14099:19 freeswitch
9195 root rt -10 9823.6m 3.6g 7328 S 22.2 23.4 5259:52 freeswitch
9191 root rt -10 9823.6m 3.6g 7328 R 13.2 23.4 3610:41 freeswitch
9178 root rt -10 9823.6m 3.6g 7328 S 2.3 23.4 696:45.75 freeswitch
pstack 9187
Thread 1 (process 9187):
#0 0x00007f18b9a640e3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f18bb1bba9a in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#2 0x00007f18bb1b4211 in su_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#3 0x00007f18bb1b5513 in su_root_yield () from /usr/local/lib/libsofia-sip-ua.so.0
#4 0x00007f18bb14bb2c in nua_stack_timer () from /usr/local/lib/libsofia-sip-ua.so.0
#5 0x00007f18bb1b70c5 in su_timer_expire () from /usr/local/lib/libsofia-sip-ua.so.0
#6 0x00007f18bb1b7f58 in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#7 0x00007f18bb1b423c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#8 0x00007f18bb1b5316 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
#9 0x00007f18bb1b8d74 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
#10 0x00007f18ba40fea5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f18b9a63b0d in clone () from /lib64/libc.so.6
pstack 9191
Thread 1 (process 9191):
#0 0x00007f18b9a640e3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f18bb1bba9a in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#2 0x00007f18bb1b7fb8 in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#3 0x00007f18bb1b423c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#4 0x00007f18bb1b5316 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
#5 0x00007f18bb1b8d74 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
#6 0x00007f18ba40fea5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f18b9a63b0d in clone () from /lib64/libc.so.6
pstack 9195
Thread 1 (process 9195):
#0 0x00007f18b9a640e3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f18bb1bba9a in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#2 0x00007f18bb1b4211 in su_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#3 0x00007f18bb1b5513 in su_root_yield () from /usr/local/lib/libsofia-sip-ua.so.0
#4 0x00007f18bb14bb2c in nua_stack_timer () from /usr/local/lib/libsofia-sip-ua.so.0
#5 0x00007f18bb1b70c5 in su_timer_expire () from /usr/local/lib/libsofia-sip-ua.so.0
#6 0x00007f18bb1b7f58 in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#7 0x00007f18bb1b423c in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#8 0x00007f18bb1b5316 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
#9 0x00007f18bb1b8d74 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
#10 0x00007f18ba40fea5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f18b9a63b0d in clone () from /lib64/libc.so.6
Question: It shows that the epoll_wait method is causing high CPU usage. How to fix this?