HTTP API request runs into an exception during node startup

[plewam]

Hello,

I am currently playing around with rmq 3.13.0-rc.3, erlang 26.1.2.
During restart I encounter the following error log. As I had my problems with rmq streams in the past I would like to know what this process does and how I can prevent the error.

Error log

2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>   crasher:
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     initial call: cowboy_stream_h:request_process/3
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     pid: <0.7633.0>
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     registered_name: []
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     exception error: no function clause matching
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>                      proplists:get_keys(undefined,
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>                                         {set,0,16,16,8,80,48,
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>                                          {[],[],[],[],[],[],[],[],[],[],[],[],
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>                                           [],[],[],[]},
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>                                          {{[],[],[],[],[],[],[],[],[],[],[],
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>                                            [],[],[],[],[]}}}) (proplists.erl, line 336)
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>       in function  proplists:get_keys/1 (proplists.erl, line 334)
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>       in call from rabbit_mgmt_wm_nodes:all_nodes_raw/0 (rabbit_mgmt_wm_nodes.erl, line 52)
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>       in call from rabbit_mgmt_wm_nodes:all_nodes/1 (rabbit_mgmt_wm_nodes.erl, line 44)
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>       in call from rabbit_mgmt_wm_nodes:to_json/2 (rabbit_mgmt_wm_nodes.erl, line 29)
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>       in call from cowboy_rest:call/3 (src/cowboy_rest.erl, line 1583)
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>       in call from cowboy_rest:set_resp_body/2 (src/cowboy_rest.erl, line 1472)
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>       in call from cowboy_rest:upgrade/4 (src/cowboy_rest.erl, line 284)
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     ancestors: [<0.7619.0>,<0.7531.0>,<0.7528.0>,<0.7527.0>,<0.7525.0>,
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>                   rabbit_web_dispatch_sup,<0.7319.0>]
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     message_queue_len: 0
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     messages: []
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     links: [<0.7619.0>]
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     dictionary: []
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     trap_exit: false
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     status: running
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     heap_size: 4185
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     stack_size: 28
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>     reductions: 1628
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0>   neighbours:
2023-12-21 08:25:40.458000+00:00 [error] <0.7633.0> 
2023-12-21 08:25:40.458000+00:00 [error] <0.7619.0> Ranch listener {acceptor,{0,0,0,0},15672}, connection process <0.7619.0>, stream 5 had its request process <0.7633.0> exit with reason function_clause and stacktrace [{proplists,get_keys,[undefined,{set,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}],[{file,"proplists.erl"},{line,336}]},{proplists,get_keys,1,[{file,"proplists.erl"},{line,334}]},{rabbit_mgmt_wm_nodes,all_nodes_raw,0,[{file,"rabbit_mgmt_wm_nodes.erl"},{line,52}]},{rabbit_mgmt_wm_nodes,all_nodes,1,[{file,"rabbit_mgmt_wm_nodes.erl"},{line,44}]},{rabbit_mgmt_wm_nodes,to_json,2,[{file,"rabbit_mgmt_wm_nodes.erl"},{line,29}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,1583}]},{cowboy_rest,set_resp_body,2,[{file,"src/cowboy_rest.erl"},{line,1472}]},{cowboy_rest,upgrade,4,[{file,"src/cowboy_rest.erl"},{line,284}]}]
2023-12-21 08:25:40.458000+00:00 [error] <0.7619.0> 
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>   crasher:
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     initial call: cowboy_stream_h:request_process/3
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     pid: <0.7634.0>
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     registered_name: []
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     exception error: no function clause matching
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>                      proplists:get_keys(undefined,
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>                                         {set,0,16,16,8,80,48,
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>                                          {[],[],[],[],[],[],[],[],[],[],[],[],
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>                                           [],[],[],[]},
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>                                          {{[],[],[],[],[],[],[],[],[],[],[],
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>                                            [],[],[],[],[]}}}) (proplists.erl, line 336)
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>       in function  proplists:get_keys/1 (proplists.erl, line 334)
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>       in call from rabbit_mgmt_wm_nodes:all_nodes_raw/0 (rabbit_mgmt_wm_nodes.erl, line 52)
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>       in call from rabbit_mgmt_wm_nodes:all_nodes/1 (rabbit_mgmt_wm_nodes.erl, line 44)
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>       in call from rabbit_mgmt_wm_overview:web_contexts/1 (rabbit_mgmt_wm_overview.erl, line 123)
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>       in call from rabbit_mgmt_wm_overview:to_json/2 (rabbit_mgmt_wm_overview.erl, line 68)
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>       in call from cowboy_rest:call/3 (src/cowboy_rest.erl, line 1583)
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>       in call from cowboy_rest:set_resp_body/2 (src/cowboy_rest.erl, line 1472)
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     ancestors: [<0.7619.0>,<0.7531.0>,<0.7528.0>,<0.7527.0>,<0.7525.0>,
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>                   rabbit_web_dispatch_sup,<0.7319.0>]
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     message_queue_len: 0
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     messages: []
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     links: [<0.7619.0>]
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     dictionary: []
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     trap_exit: false
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     status: running
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     heap_size: 6772
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     stack_size: 28
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>     reductions: 4084
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0>   neighbours:
2023-12-21 08:25:40.458000+00:00 [error] <0.7634.0> 
2023-12-21 08:25:40.458000+00:00 [error] <0.7619.0> Ranch listener {acceptor,{0,0,0,0},15672}, connection process <0.7619.0>, stream 6 had its request process <0.7634.0> exit with reason function_clause and stacktrace [{proplists,get_keys,[undefined,{set,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}],[{file,"proplists.erl"},{line,336}]},{proplists,get_keys,1,[{file,"proplists.erl"},{line,334}]},{rabbit_mgmt_wm_nodes,all_nodes_raw,0,[{file,"rabbit_mgmt_wm_nodes.erl"},{line,52}]},{rabbit_mgmt_wm_nodes,all_nodes,1,[{file,"rabbit_mgmt_wm_nodes.erl"},{line,44}]},{rabbit_mgmt_wm_overview,web_contexts,1,[{file,"rabbit_mgmt_wm_overview.erl"},{line,123}]},{rabbit_mgmt_wm_overview,to_json,2,[{file,"rabbit_mgmt_wm_overview.erl"},{line,68}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,1583}]},{cowboy_rest,set_resp_body,2,[{file,"src/cowboy_rest.erl"},{line,1472}]}]
2023-12-21 08:25:40.458000+00:00 [error] <0.7619.0> 

[michaelklishin]

It responds to an HTTP API request, namely rabbit_mgmt_wm_overview responds to GET /api/overview requests and rabbit_mgmt_wm_nodes responds to GET /api/nodes.

S = rabbit_db_cluster:cli_cluster_status(),
Nodes = proplists:get_value(nodes, S),
Types = proplists:get_keys(Nodes),

is the code snippet in question and Nodes ends up being undefined.

We cannot suggest anything beyond that given a single stack trace. My best guess is that something has tried performing an HTTP API request on a cluster that's still in the process of being formed.