rabbit_mqtt_connection_sup terminated with reason: reached_max_restart_intensity

[wmfxly]

I used rabbitmq mqtt protocol to test a large number of connections, and found that the connection was disconnected. The log information is as follows：

rabbitmq environment info:
[root@localhost rabbitmq]# rabbitmqctl status
Status of node rabbit@localhost ...
Runtime

OS PID: 26254
OS: Linux
Uptime (seconds): 485
RabbitMQ version: 3.8.5
Node name: rabbit@localhost
Erlang configuration: Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64] [hipe]
Erlang processes: 2298 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

• rabbitmq_mqtt
• rabbitmq_management
• amqp_client
• rabbitmq_web_dispatch
• cowboy
• cowlib
• rabbitmq_management_agent

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@localhost
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@localhost/quorum/rabbit@localhost

Config files

• /etc/rabbitmq/rabbitmq.conf

Log file(s)

• /var/log/rabbitmq/[email protected]
• /var/log/rabbitmq/rabbit@localhost_upgrade.log

Alarms

(none)

Memory

Calculation strategy: rss
Memory high watermark setting: 0.6 of available memory, computed to: 4.9208 gb
allocated_unused: 0.1927 gb (57.17 %)
other_proc: 0.0619 gb (18.35 %)
code: 0.027 gb (8.01 %)
plugins: 0.0158 gb (4.69 %)
other_system: 0.0147 gb (4.36 %)
quorum_queue_procs: 0.0096 gb (2.84 %)
quorum_ets: 0.0052 gb (1.53 %)
binary: 0.0037 gb (1.1 %)
other_ets: 0.0034 gb (1.01 %)
atom: 0.0016 gb (0.46 %)
mgmt_db: 0.0006 gb (0.17 %)
connection_other: 0.0005 gb (0.14 %)
metrics: 0.0003 gb (0.09 %)
mnesia: 0.0001 gb (0.03 %)
queue_procs: 0.0001 gb (0.02 %)
msg_index: 0.0001 gb (0.02 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)

File Descriptors

Total: 6, limit: 32671
Sockets: 2, limit: 29401

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 91.2671 gb

Totals

Connection count: 0
Queue count: 3
Virtual host count: 2

Listeners

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 1883, protocol: mqtt, purpose: MQTT

rabbitmq environment info:

[root@localhost rabbitmq]# rabbitmqctl environment
Application environment of node rabbit@localhost ...
[{amqp_client,
[{prefer_ipv6,false},{ssl_options,[]},{writer_gc_threshold,1000000000}]},
{asn1,[]},
{aten,
[{detection_threshold,0.99},
{heartbeat_interval,100},
{poll_interval,1000}]},
{compiler,[]},
{cowboy,[]},
{cowlib,[]},
{credentials_obfuscation,[{enabled,true}]},
{crypto,[{fips_mode,false},{rand_cache_size,896}]},
{cuttlefish,[]},
{gen_batch_server,[]},
{goldrush,[]},
{inets,[]},
{jsx,[]},
{kernel,
[{inet_default_connect_options,[{nodelay,true}]},
{inet_dist_listen_max,25672},
{inet_dist_listen_min,25672},
{logger,
[{handler,default,logger_std_h,
#{config => #{type => standard_io},
formatter =>
{logger_formatter,
#{legacy_header => true,single_line => false}}}}]},
{logger_level,notice},
{logger_sasl_compatible,false},
{shutdown_func,{rabbit_prelaunch,shutdown_func}}]},
{lager,
[{async_threshold,20},
{async_threshold_window,5},
{colored,false},
{colors,
[{debug,"\e[0;38m"},
{info,"\e[1;37m"},
{notice,"\e[1;36m"},
{warning,"\e[1;33m"},
{error,"\e[1;31m"},
{critical,"\e[1;35m"},
{alert,"\e[1;44m"},
{emergency,"\e[1;41m"}]},
{crash_log,"log/crash.log"},
{crash_log_count,5},
{crash_log_date,"$D0"},
{crash_log_msg_size,65536},
{crash_log_rotator,lager_rotator_default},
{crash_log_size,10485760},
{error_logger_format_raw,true},
{error_logger_hwm,5000},
{error_logger_hwm_original,50},
{error_logger_redirect,true},
{extra_sinks,
[{error_logger_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_channel_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_connection_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_feature_flags_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_federation_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_ldap_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_mirroring_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_prelaunch_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_queue_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_ra_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_shovel_lager_event,
[{handlers,[{lager_forwarder_backend,[lager_event,info]}]},
{rabbit_handlers,
[{lager_forwarder_backend,[lager_event,info]}]}]},
{rabbit_log_upgrade_lager_event,
[{handlers,
[{lager_file_backend,
[{date,[]},
{file,
"/var/log/rabbitmq/rabbit@localhost_upgrade.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,info},
{size,0}]}]},
{rabbit_handlers,
[{lager_file_backend,
[{date,[]},
{file,
"/var/log/rabbitmq/rabbit@localhost_upgrade.log"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,info},
{size,0}]}]}]}]},
{handlers,
[{lager_file_backend,
[{date,[]},
{file,"/var/log/rabbitmq/[email protected]"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,debug},
{size,0}]}]},
{log_root,"/var/log/rabbitmq"},
{rabbit_handlers,
[{lager_file_backend,
[{date,[]},
{file,"/var/log/rabbitmq/[email protected]"},
{formatter_config,
[date," ",time," ",color,"[",severity,"] ",
{pid,[]},
" ",message,"\n"]},
{level,debug},
{size,0}]}]}]},
{mnesia,[{dir,"/var/lib/rabbitmq/mnesia/rabbit@localhost"}]},
{observer_cli,[{plugins,[]}]},
{os_mon,
[{start_cpu_sup,false},
{start_disksup,false},
{start_memsup,false},
{start_os_sup,false}]},
{public_key,[]},
{ra,[{data_dir,"/var/lib/rabbitmq/mnesia/rabbit@localhost/quorum"},
{logger_module,rabbit_log_ra_shim},
{wal_max_batch_size,4096},
{wal_max_size_bytes,536870912}]},
{rabbit,
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['PLAIN','AMQPLAIN']},
{autocluster,
[{peer_discovery_backend,rabbit_peer_discovery_classic_config}]},
{autoheal_state_transition_timeout,60000},
{background_gc_enabled,false},
{background_gc_target_interval,60000},
{backing_queue_module,rabbit_priority_queue},
{channel_max,128},
{channel_operation_timeout,15000},
{channel_tick_interval,60000},
{cluster_keepalive_interval,10000},
{cluster_nodes,{[],disc}},
{cluster_partition_handling,ignore},
{collect_statistics,fine},
{collect_statistics_interval,60000},
{config_entry_decoder,[{passphrase,undefined}]},
{connection_max,infinity},
{credit_flow_default_credit,{400,200}},
{default_consumer_prefetch,{false,0}},
{default_permissions,[<<".">>,<<".">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,50000000},
{disk_monitor_failure_retries,10},
{disk_monitor_failure_retry_interval,120000},
{enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
{feature_flags_file,
"/var/lib/rabbitmq/mnesia/rabbit@localhost-feature_flags"},
{fhc_read_buffering,false},
{fhc_write_buffering,true},
{frame_max,131072},
{halt_on_upgrade_failure,true},
{handshake_timeout,10000},
{heartbeat,60},
{hipe_compile,false},
{hipe_modules,
[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
rabbit_guid,rabbit_net,rabbit_amqqueue_process,
rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
rabbit_limiter,gb_trees,rabbit_queue_index,
rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
{lager_default_file,"/var/log/rabbitmq/[email protected]"},
{lager_log_root,"/var/log/rabbitmq"},
{lager_upgrade_file,"/var/log/rabbitmq/rabbit@localhost_upgrade.log"},
{lazy_queue_explicit_gc_run_operation_threshold,1000},
{log,
[{file,[{file,"/var/log/rabbitmq/[email protected]"}]},
{categories,
[{upgrade,
[{file,
"/var/log/rabbitmq/rabbit@localhost_upgrade.log"}]}]}]},
{loopback_users,[]},
{max_message_size,134217728},
{memory_monitor_interval,2500},
{mirroring_flow_control,true},
{mirroring_sync_batch_size,4096},
{mnesia_table_loading_retry_limit,10},
{mnesia_table_loading_retry_timeout,30000},
{msg_store_credit_disc_bound,{4000,800}},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{msg_store_io_batch_size,4096},
{msg_store_shutdown_timeout,600000},
{num_ssl_acceptors,10},
{num_tcp_acceptors,65535},
{password_hashing_module,rabbit_password_hashing_sha256},
{plugins_dir,
"/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.8.5/plugins"},
{plugins_expand_dir,
"/var/lib/rabbitmq/mnesia/rabbit@localhost-plugins-expand"},
{proxy_protocol,false},
{queue_explicit_gc_run_operation_threshold,1000},
{queue_index_embed_msgs_below,4096},
{queue_index_max_journal_entries,32768},
{quorum_cluster_size,5},
{quorum_commands_soft_limit,32},
{reverse_dns_lookups,false},
{server_properties,[]},
{ssl_allow_poodle_attack,false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{ssl_cert_login_from,distinguished_name},
{ssl_handshake_timeout,5000},
{ssl_listeners,[]},
{ssl_options,[]},
{tcp_listen_options,
[{sndbuf,196608},{recbuf,196608},{nodelay,true},{backlog,16384}]},
{tcp_listeners,[{"auto",5672}]},
{trace_vhosts,[]},
{vhost_restart_strategy,continue},
{vm_memory_calculation_strategy,rss},
{vm_memory_high_watermark,0.6},
{vm_memory_high_watermark_paging_ratio,0.5},
{writer_gc_threshold,1000000000}]},
{rabbit_common,[]},
{rabbitmq_management,
[{content_security_policy,
"script-src 'self' 'unsafe-eval' 'unsafe-inline'; object-src 'self'"},
{cors_allow_origins,[]},
{cors_max_age,1800},
{http_log_dir,none},
{load_definitions,none},
{management_db_cache_multiplier,5},
{process_stats_gc_timeout,300000},
{stats_event_max_backlog,250}]},
{rabbitmq_management_agent,
[{rates_mode,basic},
{sample_retention_policies,
[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
{basic,[{605,5},{3600,60}]},
{detailed,[{605,5}]}]}]},
{rabbitmq_mqtt,
[{allow_anonymous,true},
{default_user,<<"guest">>},
{exchange,<<"amq.topic">>},
{num_ssl_acceptors,10},
{num_tcp_acceptors,800},
{prefetch,10},
{proxy_protocol,false},
{retained_message_store,rabbit_mqtt_retained_msg_store_dets},
{retained_message_store_dets_sync_interval,2000},
{sparkplug,false},
{ssl_cert_login,false},
{ssl_listeners,[]},
{subscription_ttl,86400000},
{tcp_listen_options,[{sndbuf,196608},{recbuf,196608},{backlog,16384}]},
{tcp_listeners,[1883]},
{vhost,<<"/">>}]},
{rabbitmq_prelaunch,[]},
{rabbitmq_web_dispatch,[]},
{ranch,[]},
{recon,[]},
{sasl,[{errlog_type,error},{sasl_error_logger,false}]},
{ssl,[]},
{stdlib,[]},
{stdout_formatter,[]},
{syntax_tools,[]},
{sysmon_handler,
[{busy_dist_port,true},
{busy_port,false},
{gc_ms_limit,0},
{heap_word_limit,0},
{port_limit,100},
{process_limit,100},
{schedule_ms_limit,0}]},
{tools,[{file_util_search_methods,[{[],[]},{"ebin","esrc"},{"ebin","src"}]}]},
{xmerl,[]}]

error detailed log:
2021-03-05 17:17:27.921 [error] <0.26704.1> ** Generic server <0.26704.1> terminating
** Last message in was {tcp,#Port<0.5006>,<<16,63,0,6,77,81,73,115,100,112,3,194,1,44,0,35,108,111,99,97,108,104,111,115,116,95,98,101,110,99,104,95,99,111,110,110,95,51,51,56,95,51,56,54,49,56,55,56,48,49,48,0,5,97,100,109,105,110,0,5,97,100,109,105,110>>}
** When Server state == {state,#Port<0.5006>,"172.16.200.29:40909 -> 172.16.200.26:1883",true,undefined,false,running,{none,none},<0.26703.1>,false,none,{proc_state,#Port<0.5006>,#{},{undefined,undefined},{0,nil},{0,nil},undefined,1,undefined,undefined,undefined,{undefined,undefined},undefined,<<"amq.topic">>,{amqp_adapter_info,{0,0,0,0,0,65535,44048,51226},1883,{0,0,0,0,0,65535,44048,51229},40909,<<"172.16.200.29:40909 -> 172.16.200.26:1883">>,{'MQTT',"N/A"},[{channels,1},{channel_max,1},{frame_max,0},{client_properties,[{<<"product">>,longstr,<<"MQTT client">>}]},{ssl,false}]},none,undefined,undefined,#Fun<rabbit_mqtt_processor.0.78125601>,{0,0,0,0,0,65535,44048,51229},#Fun<rabbit_mqtt_util.4.100928214>,#Fun<rabbit_mqtt_util.5.100928214>,undefined},undefined,{state,fine,60000,undefined}}
** Reason for termination ==
** {{function_clause,[{amqp_gen_connection,terminate,[{timeout,{gen_server,call,[credentials_obfuscation_svc,{decrypt,{encrypted,<<"bw4k1HZxj6pF44rBxWv4wFkfKCpRCVtJtc5XIE7DBpEwm2gfYYw6urbCzMZluSDG">>}}]}},{<0.32379.1>,{amqp_params_direct,<<"admin">>,{encrypted,<<"bw4k1HZxj6pF44rBxWv4wFkfKCpRCVtJtc5XIE7DBpEwm2gfYYw6urbCzMZluSDG">>},<<"/">>,rabbit@localhost,{amqp_adapter_info,{0,0,0,0,0,65535,44048,51226},1883,{0,0,0,0,0,65535,44048,51229},40909,<<"172.16.200.29:40909 -> 172.16.200.26:1883">>,{'MQTT',"3.1.0"},[{variable_map,#{<<"client_id">> => <<"localhost_bench_conn_338_3861878010">>}},{channels,1},{channel_max,1},{frame_max,0},{client_properties,[{<<"product">>,longstr,<<"MQTT client">>},{client_id,longstr,<<"localhost_bench_conn_338_3861878010">>}]},{ssl,false}]},[]}}],[{file,"src/amqp_gen_connection.erl"},{line,242}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,673}]},{gen_server,terminate,10,[{file,"gen_server.erl"},{line,858}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{gen_server,call,[<0.32380.1>,connect,60000]}}
2021-03-05 17:17:27.921 [error] <0.26704.1> CRASH REPORT Process <0.26704.1> with 0 neighbours exited with reason: {{function_clause,[{amqp_gen_connection,terminate,[{timeout,{gen_server,call,[credentials_obfuscation_svc,{decrypt,{encrypted,<<"bw4k1HZxj6pF44rBxWv4wFkfKCpRCVtJtc5XIE7DBpEwm2gfYYw6urbCzMZluSDG">>}}]}},{<0.32379.1>,{amqp_params_direct,<<"admin">>,{encrypted,<<"bw4k1HZxj6pF44rBxWv4wFkfKCpRCVtJtc5XIE7DBpEwm2gfYYw6urbCzMZluSDG">>},<<"/">>,rabbit@localhost,{amqp_adapter_info,{0,0,0,0,0,65535,44048,51226},1883,{0,0,0,0,0,65535,44048,51229},40909,<<"172.16.200.29:40909 -> 172.16.200.26:1883">>,{...},...},...}}],...},...]},...} in gen_server2:terminate/3 line 1183
2021-03-05 17:17:27.922 [error] <0.26702.1> Supervisor {<0.26702.1>,rabbit_mqtt_connection_sup} had child rabbit_mqtt_reader started with rabbit_mqtt_reader:start_link(<0.26703.1>, {acceptor,{0,0,0,0,0,0,0,0},1883}) at <0.26704.1> exit with reason {{function_clause,[{amqp_gen_connection,terminate,[{timeout,{gen_server,call,[credentials_obfuscation_svc,{decrypt,{encrypted,<<"bw4k1HZxj6pF44rBxWv4wFkfKCpRCVtJtc5XIE7DBpEwm2gfYYw6urbCzMZluSDG">>}}]}},{<0.32379.1>,{amqp_params_direct,<<"admin">>,{encrypted,<<"bw4k1HZxj6pF44rBxWv4wFkfKCpRCVtJtc5XIE7DBpEwm2gfYYw6urbCzMZluSDG">>},<<"/">>,rabbit@localhost,{amqp_adapter_info,{0,0,0,0,0,65535,44048,51226},1883,{0,0,0,0,0,65535,44048,51229},40909,<<"172.16.200.29:40909 -> 172.16.200.26:1883">>,{...},...},...}}],...},...]},...} in context child_terminated
2021-03-05 17:17:27.922 [error] <0.26702.1> Supervisor {<0.26702.1>,rabbit_mqtt_connection_sup} had child rabbit_mqtt_reader started with rabbit_mqtt_reader:start_link(<0.26703.1>, {acceptor,{0,0,0,0,0,0,0,0},1883}) at <0.26704.1> exit with reason reached_max_restart_intensity in context shutdown

crash log:
2021-03-05 17:12:28 =SUPERVISOR REPORT====
Supervisor: {<0.27643.0>,amqp_channel_sup_sup}
Context: shutdown_error
Reason: shutdown
Offender: [{nb_children,1},{name,channel_sup},{mfargs,{amqp_channel_sup,start_link,[direct,<0.22011.0>,<<"172.16.200.29:58112 -> 172.16.200.26:1883">>]}},{restart_type,temporary},{shutdown,infinity},{child_type,supervisor}]

2021-03-05 17:12:28 =SUPERVISOR REPORT====
Supervisor: {<0.1633.1>,amqp_channel_sup_sup}
Context: shutdown_error
Reason: shutdown
Offender: [{nb_children,1},{name,channel_sup},{mfargs,{amqp_channel_sup,start_link,[direct,<0.28801.0>,<<"172.16.200.29:59054 -> 172.16.200.26:1883">>]}},{restart_type,temporary},{shutdown,infinity},{child_type,supervisor}]

Looking forward to reply, any helpful information. thank you very much.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

Connections repeatedly fail at a high rate because obfuscated credentials in connection state runs into a timeout while decrypting.

There was at least one very important fix to the MQTT plugin since 3.8.5. You are 9 patch releases and 9 months out of date. Please upgrade.

Specifically relevant issues that I can quickly see in the change logs since 3.8.5:

• Revert two commits for #233 rabbitmq-mqtt#239
• Coerce password to a binary rabbitmq-federation#112 (which caused credentials_obfuscation changes potentially relevant here)
• Optimise MQTT state machine #2692 (relevant to every MQTT Plugin user, not necessarily here)

[wmfxly]

Perfect, I upgraded my rabbitmq to 3.8.14, which solved the concurrent connection problem. thank you very much. But I still have a small problem, I want to test the persistence of rabbimtmq mqtt. But I found the following information in the official website: https://www.rabbitmq.com/mqtt.html

These implementations are suitable for development but sometimes won't be for production needs. MQTT 3.1 specification does not define consistency or replication requirements for retained message stores, therefore RabbitMQ allows for custom ones to meet the consistency and availability needs of a particular environment. For example, stores based on Riak and Cassandra would be suitable for most production environments as those data stores provide tunable consistency.

I want custom ones to meet the consistency and availability needs of a particular environment For example Cassandra. Is there a good way to solve this problem.

Looking forward to reply, any helpful information. thank you very much.

[michaelklishin]

Also worth mentioning here since your test very likely opens connections at a very high rate:

• Why high connection churn is problematic
• Tuning for a large number of concurrent connections

They are equally applicable to all supported protocols.

[dormanze]

My version of rabbitmq is : RabbitMQ 3.8.22 , There are also such problems.

[michaelklishin]

I'm not sure how 35% of memory footprint in "allocated but unused" is relevant in this discussion. This is very likely due to memory fragmentation. Prometheus and Grafana provide extra dashboards for runtime metrics.

Both Erlang 23 and RabbitMQ 3.8 have reached end-of-life.