Timeout when use rabbitmqadmin to declare parameters

[SunHaoran-1st]

Hi all, hope somebody can help on this one :)

Basic info:

1. RabbitMQ Version: 3.9.14
2. Erlang Version: 23.2
3. Clusters: node1, node2, node3, node4
4. System: Centos7.8

All steps are performed at node 4.
Here is my steps:

1. rabbitmqadmin declare parameter component=federation-upstream name=upstream.1.ex value='{"ack-mode": "on-confirm", "exchange": "pub.ex", "expires": 1000, "max-hops": 2, "reconnect-delay": 1, "trust-user-id": false, "uri": "amqp://guest:guest@[nodes1 ip]:port"}'

2. declare upstream.2.ex and upstream.3.ex for nodes2 and nodes3

3. rabbitmqadmin declare parameter component=federation-upstream-set name=upstreamset.ex value='[]'

4. rabbitmqadmin declare policy name=fed-ex pattern='^sub.ex$' definition='{"federation-upstream-set": "upstreamset.ex"}' apply-to=exchanges

5. rabbitmqadmin declare parameter component=federation-upstream-set name=upstreamset.ex value='[{"upstream": "upstream.1.ex"}, {"upstream": "upstream.2.ex"}, {"upstream": "upstream.3.ex"}]' --request-timeout=1

Normally, I get parameter declared in step 5, and the federaion link is correct with upstream.1.ex,upstream.2.ex,upstream.3.ex.
But sometimes I get Timed out getting HTTP response in step5, and the federaion link is incomplete with just upstream.1.ex or upstream.1.ex,upstream.2.ex.
I try to declare upstream-set again, and get parameter declared immediately, but the federaion link do not become correct.

Here is my questions:

1. When declare upstream-set timeout, why re-declare upstream-set does not make the federaion link correct?
2. When declare upstream-set timeout, how to make the federaion link correct?

Thanks for your help.

[michaelklishin]

RabbitMQ 3.9 has reached end of life.

Something in your environment causes HTTP API operations to time out from time to time. See logs and monitoring data for clues. Declaring an upstream is a very cheap operation. Adding a policy can be very expensive because it involves updating the state of all queues in the system, starting a lot of federation links, things like that.

A very common reason for this is HTTP API-based monitoring that uses GET /api/queues which is very wasteful. Use Prometheus instead.

[SunHaoran-1st]

Thanks for your answer.

I think the HTTP API request timeouts are caused by high CPU usage on my device and network latency and --request-timeout=1 setting.
In addition, I have some other details regarding the use of federation that I would like to consult with you.

From this doc, I speculate that when both the upstream and policy remain unchanged, only PUT /api/parameters/federation-upstream-set will result in different handling methods.

• upstream changed in the upstream-set: reset the upstream-set conf and reset all federation link
• upstream unchanged in the upstream-set: return success immediately, and do not reset any federation link

So, when I first PUT /api/parameters/federation-upstream-set, upstream in the upstream-set changes from empty to upstream.1.ex,upstream.2.ex,upstream.3.ex, but as soon as rabbitmq finished setting just upstream.1.ex' s federation link, it timed out by --request-timeout=1. When I try to PUT /api/parameters/federation-upstream-set again, rabbitmq thinks that the upstream in the upstream-set has not been updated, so it returns success and does not update the federation link.

The only way to reset the federation link is DELETE /api/parameters/federation-upstream-set and PUT /api/parameters/federation-upstream-set again, or DELETE /api/policy and PUT /api/policy again.

Is my guess correct? Thanks.

[michaelklishin]

Changes to upstream sets indeed can result in links being restarted. I am certain that that and similar endpoints are not idempotent.

You are correct that deleting an upstream set or a policy will cause (relevant) federation links to stop. If you want to temporarily disable federation, you can also disable the plugin and re-enable it.

[SunHaoran-1st]

Thanks for your help. I got it!