There is a crash caused by deleting upstream and upstream-set after deleting policy

[SunHaoran-1st]

Hi,
This is some basic information:

RabbitMq version: 3.7.9
Erlang version: 21
OS: Centos7.8

I use federation with guides from rabbitmq.com, set upstream, upstream-set and policy apply-to exchange.

Most of the time, everything is ok, but sometimes, after deleting the policy, I try to delete upstream-set with those API(expected 204 No Content):
1.DELETE /api/policies/%s/%s HTTP/1.1
2.DELETE /api/parameters/federation-upstream-set/%s/%s HTTP/1.1
3.DELETE /api/parameters/federation-upstream/%s/%s HTTP/1.1
here is a crash:

2022-01-25 16:13:25.903 [error] <0.10756.0> CRASH REPORT Process <0.10756.0> with 0 neighbours exited with reason: no such process or port in call to gen_server:call(undefined, which_children, infinity) in gen_server:call/3 line 223 in gen_server:call/3 line 223
2022-01-25 16:13:25.904 [error] <0.10755.0> Ranch listener rabbit_web_dispatch_sup_15672, connection process <0.10755.0>, stream 1 had its request process <0.10756.0> exit with reason {noproc,{gen_server,call,[undefined,which_children,infinity]}} and stacktrace [{gen_server,call,3,[{file,"gen_server.erl"},{line,223}]},{rabbit_federation_link_sup,adjust,3,[{file,"src/rabbit_federation_link_sup.erl"},{line,34}]},{rabbit_federation_exchange_link_sup_sup,'-adjust/1-lc$^0/1-0-',2,[{file,"src/rabbit_federation_exchange_link_sup_sup.erl"},{line,51}]},{rabbit_federation_exchange_link_sup_sup,adjust,1,[{file,"src/rabbit_federation_exchange_link_sup_sup.erl"},{line,52}]},{rabbit_federation_parameters,adjust,1,[{file,"src/rabbit_federation_parameters.erl"},{line,80}]},{rabbit_mgmt_wm_parameter,delete_resource,2,[{file,"src/rabbit_mgmt_wm_parameter.erl"},{line,83}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,1182}]},{cowboy_rest,expect,6,[{file,"src/cowboy_rest.erl"},{line,1164}]}]

I write some logs to get the information:

This crash caused by an undefined PID was sent as a parameter to
rabbit_federation_exchange_link_sup_sup.erl line:51

But, most of the time, deleting policy first and then deleting upstream-set will not cause the crash , because the return of mirrored_supervisor:which_children(?SUPERVISOR) is an empty list, like "[]".

This problem has troubled me for a long time. I don't know the intention of the process represented by that PID, and how to avoid this crash, please give me some advices, thanks!!!

ps:
I try to change the sequence of API, delete upstream-set first, delete the policy at the end, and I find that crash never occurs. But sometime the API has no return (Expected "204 No Content").
1.DELETE /api/parameters/federation-upstream/%s/%s HTTP/1.1
2.DELETE /api/parameters/federation-upstream-set/%s/%s HTTP/1.1
3.DELETE /api/policies/%s/%s HTTP/1.1

[lukebakken]

Thank you for the detailed report. Since you can reproduce this issue, please try to do so using the latest version of RabbitMQ (3.9.13).

[SunHaoran-1st]

Thanks for you suggestion, I tried RabbitMQ (3.9.11) with Erlang(24.2), but I still got the same problem.
And I will try RabbitMQ (3.9.13) later.

[SunHaoran-1st]

I also get the same crash in RabbitMQ(3.9.11), this is a part of the crash report.

[SunHaoran-1st]

Finally, I found this crash in RabbitMQ(3.9.13).

2022-02-16 14:35:30.302856+08:00 [error] <0.27657.0> Ranch listener {acceptor,{0,0,0,0,0,0,0,0},15672}, connection process <0.27657.0>, stream 1 had its request process <0.27658.0> exit with reason {noproc,{gen_server,call,[undefined,which_children,infinity]}} and stacktrace [{gen_server,call,3,[{file,"gen_server.erl"},{line,247}]},{rabbit_federation_link_sup,adjust,3,[{file,"rabbit_federation_link_sup.erl"},{line,26}]},{rabbit_federation_exchange_link_sup_sup,'-adjust/1-lc$^1/1-0-',2,[{file,"rabbit_federation_exchange_link_sup_sup.erl"},{line,57}]},{rabbit_federation_exchange_link_sup_sup,adjust,1,[{file,"rabbit_federation_exchange_link_sup_sup.erl"},{line,58}]},{rabbit_federation_parameters,adjust,1,[{file,"rabbit_federation_parameters.erl"},{line,73}]},{rabbit_mgmt_wm_parameter,delete_resource,2,[{file,"rabbit_mgmt_wm_parameter.erl"},{line,74}]},{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,1575}]},{cowboy_rest,expect,6,[{file,"src/cowboy_rest.erl"},{line,1558}]}]

[SunHaoran-1st]

Hello lukebakken!
@lukebakken
This crash occurs probabilistically in latest version of RabbitMQ(3.9.13), maybe caused by the time between deleting the policy and deleting the upstream-set is too short. Is it a problem with my use of steps or an internal bug in the program?
If you have some idea, please give me some more advice, thanks a lot! : ）

[lukebakken]

I appreciate all of the detailed information. Please be patient, however. This crash does not appear to affect the normal operation of RabbitMQ. Someone on the RabbitMQ team will try to find time to investigate this over the next week or two.

[SunHaoran-1st]

Hello!
What is the result of your investigating, thanks.

[michaelklishin]

Nothing to report. This is open source software, so you are welcome to conduct your own investigation.