JMS plugin: selector deletion by one client causes a publishing client to be disconnected

[TomGood75]

Good afternoon!

• I've run into an issue where if a jms-message-selector exchange is destroyed by one client while another client is publishing, it triggers an error to be thrown, and the publishing client's connection to be closed.
• Apologies in advance, I'm not familiar with Erlang. If my interpretation efforts are correct, the issue lies in rabbit_jms_topic_exchange.erl. When a message comes through 'route', get 'get_binding_fun_x' is trying to read_state on the exchange which is/was being destroyed. This then causes an "exchange named [] has no saved state or incorrect saved state" from line 289 to be thrown, which in turn disconnects the publishing client. Example error, from a test case to reproduce the error I'm running into periodically (regularly) in production:

2022-04-22 15:59:20.025335+00:00 [erro] <0.28247.50> Error on AMQP connection <0.28247.50> (172.17.14.169:56946 -> 172.16.1.71:5672 - testcon, vhost: 'test', user: 'lme', state: running), channel 1:
2022-04-22 15:59:20.025335+00:00 [erro] <0.28247.50> operation basic.publish caused a connection exception internal_error: "exchange named 'jms-ndtop-slx-5e26b4e1-ed59-47fa-ba8f-502a245b6430' has no saved state or incorrect saved state"
2022-04-22 15:59:20.029623+00:00 [info] <0.28777.50> closing AMQP connection <0.28777.50> (172.17.14.169:56969 -> 172.16.1.71:5672 - testcon, vhost: 'test', user: 'lme')
2022-04-22 15:59:20.034313+00:00 [info] <0.28247.50> closing AMQP connection <0.28247.50> (172.17.14.169:56946 -> 172.16.1.71:5672 - testcon, vhost: 'test', user: 'lme')

• I'm not sure what the best way to handle this is - and am hoping some of the amazing people who maintain will. At least to me, it seems a failure to access/read-lock an exchange to process a message through it should not result in the publishing client being disconnected - especially when in this case it is due to the exchange being destroyed from another client disconnecting.

• Just let me know if you'd like any additional info. Since this only happened sporadically in production, I wrote an (ugly) test case to pin down where the issue was, which led me here once I could reproduce this action of one client with a message-selector exchange subscription disconnecting causing another (completely separate JVM client in this case) publishing to be disconnected.

• Thanks in advance for any attention you can give this issue!

[michaelklishin]

While RabbitMQ JMS selector plugin could handle such cases, I don't see how it would be meaningfully different from a case where one client deletes an exchange another one is concurrently publishing to. Since publishing to a non-existent exchange is a protocol error
(a channel one, though), the channel of that client will be closed.

We don't have any code that can be used to reproduce but the only scenario I can think of when
an entity is deleted after client disconnection is when said entity is auto-delete.
Using shared auto-delete resources is asking for trouble. Consider using named durable exchanges
if they are meant to be shared.

This is discussion material in its current state.

[TomGood75]

• As far as reproducing, I believe what I have could be boiled down to one client put into a loop publishing to a topic (jms.durable.topic exchange underlying). Client two subscribes to topic, with a message selector (any) specified - to trigger intermediary filter exchange to be created for it. When client two closes, client one ends up having the invalid exchange state protocol error thrown back to it, and is disconnected. May need to cycle client two multiple times to hit the timing right to throw the error - my test case spun up multiple processes publishing and subscribing to be able to hit the error within 30 seconds or so. Removing the message selector, the test processes could subscribe and publish indefinitely without publishers being dropped. I'd be happy to test any possible changes against what I have, or try to put together a simpler test case if this is still being viewed as a questionable issue.

[michaelklishin]

If the exchange exists, are you sure that no client intentionally deletes it? This is best detected using a traffic capture because observing internal events would only tell if you something was deleted and by what user, which may or may not be enough.

[michaelklishin]

An executable example in a public GitHub repo would be the best option. Our team's time budget is limited.

[TomGood75]

@michaelklishin Would this work for you? https://github.com/TomGood75/rmq-jms-msg-selector-test

[TomGood75]

• Sorry, I know I probably need to be more patient - but this is holding up a half-done migration project from a very old message broker to RabbitMQ, and I wanted to check back in on this in case it fell off the list.
• To add further weight/color to why I'd very much consider this a bug, and hope we can address - not only does subscriber client B disconnecting cause publishing client A to be disconnected with a protocol error - this disconnect type also blocks the client library default auto-reconnect logic. So when this occasional timing bug crops up, the only way to handle (I'm seeing so far) is manual reconnect logic in client A, as well as needing publisher confirms enabled to prevent msg loss when this happens. Due to the asynchronous nature of the protocol, usually a msg is published without error when this scenario happens before the publisher realizes it has been dropped (over client B) - so without publisher confirms, we were also losing msgs (frequently) over this.
• With the timing nature of this bug, googling of "no saved state or incorrect saved state" returning nothing (other than my 'help?' post to community support now), and the UID in the error not tying to anything helpful to troubleshoot - this caused quite a few headaches, and plenty of hours of troubleshooting and research to get to the bottom of what was going on. Since traffic is lighter in QA/test environments, we didn't run into this until heavier production use, seeing erratic connection drops (which did not reconnect) with this error. None of this your fault obviously - but it would be hugely appreciated if we could get this reconsidered as a bug to be resolved.

[TomGood75]

• I thought I should maybe turn my comment above into a proposed solution here:

Thanks! - Agreed publishing to a non-existent exchange would be a channel-level error, but in this case the client publishing is being done to the jms.durable.topic exchange - which is durable and does exist. Since the exchange being published to exists, shouldn't the publishing client be allowed to continue without error/disconnect?

It is the x-jms-topic exchange (bound to the jms.durable.topic published to) used to create a custom message selector being destroyed when client B leaves which is causing the channel disconnect on the publish to jms.durable.topic for client A.

Maybe I'm missunerstanding intent, but this seems like it would be the standard use case for the x-jms-topic exchange. The (java JMS in this case, but seems irrelevant) client subscribes to a topic. If it subscribes to the topic wide-open, the client library just creates a queue bound to the primary jms.durable.topic others will publish to, using the routing key. If a message selector is provided for the subscription, a x-jms-topic exchange is bound in between the subscriber's queue and the jms.durable.topic to apply the message filtering.

• It seems this x-jms-topic exchange being dynamically generated to filter client messages from jms.durable.topic exchange should be auto-delete, and be removed when the client subscribing closes. It does not seem this subscriber client resource closing should have the possibility to disconnect a client publishing to jms.durable.topic - which is why I initially proposed this as a bug that the x-jms-topic exchange shouldn't be throwing this back as a protocol error (or possibly any error at all?) - causing the client (channel) publishing to a valid exchange to be closed.

[michaelklishin]

Thank you. We are finishing 3.10 and introducing testing on Ubuntu 22.04 into our pipelines, so getting to this can take about a week.

[acogoluegnes]

@TomGood75 Thanks for the sample project, I managed to reproduce.

@michaelklishin My understanding is that the JMS topic exchange is deleted but some messages still manage to get in, translating to a call to route/2. As the exchange already deleted its state from mnesia, the route function returns a protocol error which bubbles up to the publisher. Reminder: when a topic subscriber with a selector is created, a queue and a JMS topic exchange are created for it, the queue is bound to the JMS topic exchange, which is itself bound to the jms.durable.topic topic exchange.

So we're indeed in a case where the consumer affects the publisher, which should not happen. There's no way here to make the difference between some invalid routing (no mnesia record for this consumer) and a legit race condition when the exchange is deleted. I don't know why the topic does not return [] in the first place, instead of failing.

I don't know the impacts it would have, but I'd vote to return [] (nowhere to route) when no record is found in Mnesia.

[michaelklishin]

@acogoluegnes OK, that sounds reasonable.

[TomGood75]

@acogoluegnes - glad to hear you were able to reproduce with the example project - I appreciate your time into this!

[acogoluegnes]

Here is the PR with the fix: #4696.

It should go into 3.10.1 and 3.9.17.

[TomGood75]

• I built RMQ from source and copied the JMS plugin to the RMQ instance I was testing against, and can no longer reproduce the issue with the above PR in play. Looks great to me, thank you guys!

[michaelklishin]

Thank you for reporting back!