Upgraded from 3.13 to 4.1.7, outgoing TLS connections to external services now fail

[Mangesh-24]

Describe the bug

Hi Team,

We have upgraded RabbitMQ version from 3.13 to 4.1.7. after that getting timeout error during handshake for protocol 'amqp/ssl' and peer #.#.#.#:34480�[0m3.

Environment details

1. RMQ hosted on Azure Kubernetes 1.30.
2. below RMQ operators and cluster version.
   Component Current Version Recommended Upgrade Version
   RabbitMQ Cluster Image rabbitmq:3.13.4-management rabbitmq:4.1.7-management
   Cluster Operator Image rabbitmqoperator/cluster-operator:2.9.0 rabbitmqoperator/cluster-operator:2.18.0
   Cluster Operator Controller v0.14.0 v0.19.0
   Messaging Topology Operator rabbitmqoperator/messaging-topology-operator:1.14.2 rabbitmqoperator/messaging-topology-operator:1.18.2
   Messaging Topology Controller v0.15.0 v0.19.0

Please check this issue and provide you guidance to resolve this issue.

Reproduction steps

1. Install RMQ cluster operators using helm chart
2. Install RMQ 3.13 version cluster using helm chart
3. There is no error TLS handshake for 3.13 version.
4. Upgrade RMQ 4.1.7 version on top of 3.13 using helm chart
5. its connections through error TLS handshake, as per investigate, RMQ pod trying to connect azure health prob pods like Prometheus etc, but not sure.
   .

Expected behavior

RMQ cluster should Upgraded from 3.13 to 4.1.7 without any issue or error and all existing resources like queues, exchanged, should work as it is.

Additional context

No response

[michaelklishin]

@Mangesh-24 our Community Support Policy explicitly states that we won't troubleshoot networking or TLS connectivity for non-paying users.

RabbitmQ 4.1.x goes out of community support in less than 48 hours on top of that.

We cannot comment on TLS failures without a certain amount of troubleshooting data.

That said, since it is an outgoing TLS connection, almost certainly this is the well known change in Erlang 26 and later versions: outgoing TLS connections now have peer verification enabled by default. So in some cases it should be disabled or the trusted CA list on the host need to be changed.

Conclusion

I don't know what part of RabbitMQ tries to connect to Azure Health Prob. We have no data to work with.

But many users going from 3.13 on Erlang 25 to 4.x on Erlang 26 or 27 run into failing outgoing TLS connections because in Erlang's TLS implementation, peer verification in the TLS client is now enabled by default.

Federation, Shovel doc guides were updated to explain how can peer verification be disabled
(for their outgoing connections). If Azure ships custom RabbitMQ plugins that connect to their
monitoring infrastructure, the must either fix peer verification (by making sure their RabbitMQ nodes consider external service CA certificates trusted) or disable it.

Please take it from here.

[michaelklishin]

This is mentioned in the 4.1.0 release notes under the Erlang/OTP Compatibility Notes section.