rabbitmq 3.8.9 start failed "Failed to create thread: Resource temporarily unavailable"

[OceanChen-GH]

Describe the bug

Install rabbitmq 3.8.9 on a server which has 512g memory

Starting rabbitmq-server: ^CFAILED - check /var/log/rabbitmq/startup_{log, _err}
rabbitmq-server.
cat /var/log/rabbitmq/startup_err
Failed to create thread: Resource temporarily unavailable (11)
/usr/lib/rabbitmq/bin/rabbitmq-server: line 182: 32656 已放弃 (core dumped) start_rabbitmq_server "$@"

Reproduction steps

1.install with a shell
#!/bin/sh
install_log="/var/log/install_rabbitmq.log"
function log () {
echo "[$1] $2"
echo "$(date "+%Y-%m-%d %H:%M:%S") [$1] $2" >> $install_log
}
#卸载rabbitmq
function uninstall () {
log "info" "卸载RabbitMQ...."
for i in rpm -qa | grep rabbitmq | grep -v grep
do
log "debug" "删除已安装的rabbitMQ -> "$i
rpm -e --nodeps $i
done

for i in `rpm -qa | grep erlang | grep -v grep`
do
    log "debug" "删除已安装的erlang -> "$i
    rpm -e --nodeps $i
done
#删除旧仓库记录
rm -rf /var/lib/rabbitmq
#删除旧日志
rm -rf /var/log/rabbitmq
#删掉lib
rm -rf /usr/lib/rabbitmq
log "info" "RabbitMQ卸载完成！"

}
#安装rabbitmq
function installmq() {
log "info" "开始安装RabbitMQ 3.7.21...."
rpm -ivh --replacefiles --nodeps logrotate-3.7.8-28.el6.x86_64.rpm
rpm -ivh --replacefiles --nodeps socat-1.7.2.3-1.el6.x86_64.rpm
#rpm -ivh --replacefiles erlang-22.1.5-1.el6.x86_64.rpm
#rpm -ivh --replacefiles rabbitmq-server-3.7.21-1.el6.noarch.rpm
rpm -ivh --replacefiles erlang-23.1.4-1.el6.x86_64.rpm
rpm -ivh --replacefiles rabbitmq-server-3.8.9-1.el6.noarch.rpm
#mq加入系统服务化
log "debug" "rabbitmq加入系统服务..."
chkconfig rabbitmq-server on
log "debug" "启动rabbit服务..."
service rabbitmq-server start
log "info" "配置rabbitmq账户信息..."
cd /usr/lib/rabbitmq/bin
rabbitmqctl add_user root xxxx
if [[ $? -ne 0 ]]; then
log "err" "添加MQ账户失败，可能是账户已存在，请稍后手动检查。"
fi
rabbitmqctl set_user_tags root administrator
rabbitmqctl set_permissions -p "/" root "." "." ".*"
log "debug" "开始rabbitmq-plugins:rabbitmq_management rabbitmq_top"
rabbitmq-plugins enable rabbitmq_management
rabbitmq-plugins enable rabbitmq_top
log "info" "RabbitMQ安装完成..."
}
#main start
uninstall
installmq

...

Expected behavior

Install rabbitmq 3.8.9 on a vm which only has 2g memory, can start success

[root@localhost ~]#
[root@localhost ~]# rabbitmqctl status
Status of node rabbit@localhost ...
Runtime

OS PID: 31983
OS: Linux
Uptime (seconds): 18488
Is under maintenance?: false
RabbitMQ version: 3.8.9
Node name: rabbit@localhost
Erlang configuration: Erlang/OTP 23 [erts-11.1.3] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:64] [hipe]
Erlang processes: 1204 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

• rabbitmq_top
• rabbitmq_management
• amqp_client
• rabbitmq_web_dispatch
• cowboy
• cowlib
• rabbitmq_management_agent

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@localhost
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@localhost/quorum/rabbit@localhost

Config files

• /etc/rabbitmq/rabbitmq.config

Log file(s)

• /var/log/rabbitmq/[email protected]
• /var/log/rabbitmq/rabbit@localhost_upgrade.log

Alarms

(none)

Memory

Total memory used: 0.0795 gb
Calculation strategy: rss
Memory high watermark setting: 4.295 gb, computed to: 0.7241 gb

other_proc: 0.0309 gb (22.35 %)
code: 0.0279 gb (20.19 %)
allocated_unused: 0.0268 gb (19.34 %)
plugins: 0.0167 gb (12.05 %)
other_system: 0.0126 gb (9.1 %)
binary: 0.0072 gb (5.2 %)
connection_other: 0.0042 gb (3.06 %)
other_ets: 0.0034 gb (2.46 %)
mgmt_db: 0.0029 gb (2.11 %)
queue_procs: 0.0019 gb (1.34 %)
atom: 0.0014 gb (1.03 %)
connection_channels: 0.0009 gb (0.66 %)
metrics: 0.0006 gb (0.42 %)
connection_readers: 0.0004 gb (0.29 %)
mnesia: 0.0003 gb (0.23 %)
connection_writers: 0.0002 gb (0.13 %)
msg_index: 0.0 gb (0.02 %)
quorum_ets: 0.0 gb (0.01 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)

File Descriptors

Total: 12, limit: 927
Sockets: 10, limit: 832

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 4.7006 gb

Totals

Connection count: 10
Queue count: 108
Virtual host count: 1

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
[root@localhost ~]#

Additional context

No response

[OceanChen-GH]

on server, ulimit -n
65535

[michaelklishin]

File handles are not relevant here. The runtime reports that it cannot start a kernel thread, not open a file handle.

[michaelklishin]

RabbitMQ 3.8 has reached a complete end of life one year ago.

[michaelklishin]

Erlang/OTP 23 [erts-11.1.3] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:64] [hipe]

suggests that the runtime inferred one core available to it but it is configured to use 64 async I/O threads.
Neither RabbitMQ nor Erlang make much sense in single core environments. You can try allowing the runtime to use 4 cores (using cgroups, VM settings or what have you) to compare.

Quick searching around for

Failed to create thread: Resource temporarily unavailable (11)

suggests resource limits on behalf of the OS, tools such as SELinux or its (likely virtual, not physical) hardware. Please take it from here.