Connection closed due to "CHANNEL_ERROR - second 'channel.open' seen"

[shubhang93]

We are constantly getting this error "com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=504, reply-text=CHANNEL_ERROR - second 'channel.open' seen, class-id=20, method-id=10)" with langohr 5.2.0. We are using a 5 node RabbitMQ cluster running version 3.9.11. We are opening a new channel for every publish, we close the channel immediately after the publish, to prevent race conditions. We saw the second 'channel.open' seen error even on the RabbitMQ broker. This error was not present when we were using the RabbitMQ server 3.8.5 with langohr 5.2.0. After upgrading to 3.9.11 we're facing this issue.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

Nothing has changed around how channels are managed in 3.9, in fact, nothing has changed in over five years.

This error means that you try to open a channel with the same number (ID) after it's been already open. This is not an issue
in either RabbitMQ or the Java client (and Langhor does not manage channels in any way).

You must be sharing a resource (such as a connection)( across threads, intentionally or not.

We highly recommend against opening a channel for every message published as it absolutely ruins publishing throughput, does not help one bit with race conditions (or I cannot think of where it would be) and creates a high connection churn scenario.

Opening channels all the time across threads can wreck chaos to the channel allocator on your connection, I cannot
immediately tell if it synchronises most of the operations involved but if it does not, it's because that's not how connections
are supposed to be used.

[mjayprateek]

Hi @michaelklishin
I'm from the same team as @shubhang93 (OP).

Though I agree with everything which you've said and we're making those optimizations in our code but we're seeing a weird behaviour with respect to "second 'channel.open' see" errors.

We're seeing these errors very frequently on RabbitMQ cluster which uses 3.9.11 but the same code if being used with the cluster which runs 3.8.5, doesn't receive such exceptions.

What could have changed between 3.8.5 to 3.9.11 ? A stricter implementation of AMQP 0-9-1 perhaps? Or, is it because we haven't setup the cluster properly? what else could be the reason here?

We need to know this because the stability of our rabbitmq workers is getting affected.

We're introducing a channel-pool to reduce the channel churn (in a hope that above errors will get eliminated) - would this be sufficient or should we move back to RabbitMQ 3.8.5?

[mjayprateek]

But, how do we know the root cause here. With the exact same code, why is it not happening on 3.8.5 but for 3.9.11?

Is there any command, diagnostic tools, configs you could help us with which would help us get to the root.

[mkuratczyk]

If you want to find the root cause, the most straightforward way is a binary search to find the version that changed the behavior - try 3.9.0 and 3.8.29 to see if it was introduced in 3.9. If your app fails with these versions - try something in the middle between 3.8.5 and most recent 3.8 and so on. Identifying which version changed the behavior should make it easy to point to a specific change.

[lukebakken]

@mjayprateek if you could do what @mkuratczyk suggests it would help out the RabbitMQ team immensely! Thanks!

[lukebakken]

More than likely there is a bug in your code or the library you are using that a newer version of RabbitMQ uncovers.

Now we maintain two connections one for production and one of consumption but this code is not in production yet. Could this have happened because we are using the same connection for both production and consumption?

Maybe. Though if you change multiple things at once finding the root cause will be more difficult.

Is the channel ID / Number assigned by the broker or the client itself generates and ID / Number

It is assigned by the client.

[michaelklishin]

The switch to a single connection may have introduced a concurrent access scenario that did not exist previously. We recommend separate connections for unrelated reasons: this way no consumer-related operations would be blocked in case of a resource alarm.