Federation with conflicting user permissions

[barkofdelight]

I'm seeing behavior I don't expect when using Federation with user permissions.

On RabbitB I create an upstream to RabbitA using UserA's credentials. On RabbitA, UserA does not have full permissions. (In my test UserA was limited to a specific Topic Read regexp.) UserB on RabbitB is not restricted.

When a pika client connects to RabbitB as UserB and binds to the forwarded exchange, all works well if the routing key is allowed for UserA on RabbitA. But what is expected when the routing key is not allowed?

I expected the client to report an error as it would when connecting directly to RabbitA as UserA. But what happens is that the client thinks everything has worked! Meanwhile, the upstream fails and starts repeatedly restarting.

Since the upstream is failing, other clients of RabbitB relying on that upstream no longer receive messages.

I don't see much documentation about using Federation along with user permissions. What is the expected behavior?

Thanks.

[RabbitMQ 3.9.11, Erlang 24.2]

[michaelklishin]

Permissions work for federation links the same way they do for other clients. However, federation links only have one approach to reacting to errors: they fail and restart. An upstream cannot "fail", not any more than a channel or queue replica can fail. It can simply reject certain client operations.

Federation links won't propagate channel errors to clients. Therefore it is highly advisable to use users with sufficient permissions so that links do not run into permission violations often. The only way to find that out would be from the logs.

[barkofdelight]

What you say makes sense, but the reason I used the phrase "the upstream fails" is because in my test it is not merely "rejecting certain client operations". After a client attempts to bind using a disallowed routing key, no more messages are being delivered through that upstream connection to any connected clients. As long as the errant bind is connected, the upstream is effectively disconnected.

I understand that the particular bind using the disallowed key won't get any messages (although I'm still surprised errors aren't propagated), but it seems quite counter-intuitive for other messages through that upstream to stop.

Thanks.

[michaelklishin]

I find it very hard to believe that a permission violation of one publisher can affect others. AuthN and authZ exceptions are scoped to channels. Worst case an entire connection can stop after a channel exceptions (that's how Shovel and Federation links behave).

Please share logs and a way to reproduce or we won't be able to help. We do not guess in this community.

[barkofdelight]

Perhaps I should have used the phrase "Upstream Link" instead of merely "Upstream".

If indeed, for Federation links, "an entire connection can stop after a channel exceptions", then perhaps what I'm seeing is expected behavior.

But just in case, I will describe in detail my test setup. (If this is not sufficient to reproduce I can provide scripts.)

When a disallowed routing_key is used, I see a restart loop for that particular Upstream Link. If a 2nd client connects on a different exchange, a 2nd Link is created (I'm seeing this on the web management page under "Federation Status/Running Links"). Such a client works properly. But an additional Client on the same exchange (using a different routing key) does not cause a separate Link to be created and that client does not receive messages until the offending client disconnects.

Setup is:

RabbitF1:
guest:
Administrator
user1:
Topic Permissions:
amq.topic: .test.

RabbitF2:
guest:
Administrator
Federation Upstream:
RabbitF1:
URI: amqp://user1:user1@RabbitF1
Policies:
amq.topic, federation-upstream-set:all
amp.fanout, federation-upstream-set:all

I then start two listening processes (using pika in python):

Listener1: guest@RabbitF2, amq.topic, routing_key *.test.M1
Listener2: guest@RabbitF2, amq.fanout

I then start two sending processes:

Sender1: guest@RabbitF1, amq.topic, routing_key A.test.M1
Sender2: guest@RabbitF1, amq.fanout

Messages are delivered properly.

Then, start a 3rd listening process:

Listener3: guest@RabbitF2, amq.topic, routing_key *.text.M1

[Note 'text' instead of 'test'.]
Listener3 does not get any messages, Listener 2 continues to get messages. The part that surprised me was the Listener1 stops getting messages until Listener3 is terminated.

An excerpt from the log file is attached. What I've copied there repeats over and over.
logExcerpt.txt

Thanks.