How connections are closed when a disk alarm is in effect

[johanrhodin]

Set the disk limit to something larger than the disk space available:
rabbitmqctl set_disk_free_limit 1000GB

Create a connection, channel (with confirm) and start publishing (see attached script for an example, also reproduced with ruby bunny)

Forcefully stop the client application (ctrl-C)

The connection will now be removed (when checking ss or netstat) but the connection is still in blocked state in RabbitMQ Management and rabbitmqctl list_connections still shows it. (rabbitmqctl eval 'inet:i().' does not show the connection, but some state is left over in the ports as seen with observer)

Once the alarm is lifted all connection info is released and logged at the same time:

2022-04-25 20:12:44.667082+00:00 [erro] <0.15157.53> closing AMQP connection <0.15157.53> (CLIENT:8862 -> 10.56.72.82:5671 - a7160cd0c6a704bd87a479c3caed0e):
2022-04-25 20:12:44.667082+00:00 [erro] <0.15157.53> {inet_error,closed}
2022-04-25 20:12:44.667079+00:00 [erro] <0.14089.53> closing AMQP connection <0.14089.53> (CLIENT:12692 -> 10.56.72.82:5671 - 7d84b7aedae8e4ce796dc5acc1f310):
2022-04-25 20:12:44.667079+00:00 [erro] <0.14089.53> {inet_error,closed}
2022-04-25 20:12:44.667223+00:00 [erro] <0.12851.53> closing AMQP connection <0.12851.53> (CLIENT:6512 -> 10.56.72.82:5671 - bd1c26c3db292d33f8d93f8e86a2b2):
2022-04-25 20:12:44.667223+00:00 [erro] <0.12851.53> {inet_error,closed}
2022-04-25 20:12:44.667349+00:00 [erro] <0.14629.53> closing AMQP connection <0.14629.53> (CLIENT:65283 -> 10.56.72.82:5671 - ae910354788f1c8ce81899bde3962b):
2022-04-25 20:12:44.667349+00:00 [erro] <0.14629.53> {inet_error,closed}
2022-04-25 20:12:44.667447+00:00 [erro] <0.14739.53> closing AMQP connection <0.14739.53> (CLIENT:7194 -> 10.56.72.82:5671 - c2d7d7d6e5f8322b4c132ef5b400fe):
2022-04-25 20:12:44.667447+00:00 [erro] <0.14739.53> {inet_error,closed}
2022-04-25 20:12:44.667544+00:00 [erro] <0.15509.53> closing AMQP connection <0.15509.53> (CLIENT:24006 -> 10.56.72.82:5671 - 3dbc61cb9998feb56579663c8959d7):
2022-04-25 20:12:44.667544+00:00 [erro] <0.15509.53> {inet_error,closed}

[michaelklishin]

When a connection is in alarmed state, it stops reading from the socket. When it stops reading from the socket, there are certain events that it won't be able easily observe, connection closure
by the peer is one of such things. A TCP connection closure is initiated by one peer
but it ultimately involves both or a significant enough timeout.

Around 2015-2016 Mirantis engineers managed to solve this in the only way we found to get
the job done: using heartbeat monitoring on the server. Here are the PRs I could dig up:

• Fix cluster-wide alarm handling #395
• Detect connections dropped during resource alarm rabbitmq-common#31

In environments where heartbeats are not used, we are not aware of a solution that would allow
connections to detect that the peer has gone away. CloudAMQP previously reported that they
do not rely or even outright disable heartbeats and rely on TCP keepalives instead. Then
the behavior would be exactly as it was prior to the above PRs.

We are not aware of any relevant changes to Erlang's server socket API that would make
it easy to reliably detect peer disconnection while at the same time not reading from the socket.

[johanrhodin]

I think I reproduced this with heartbeats on, let me double check tomorrow.

[michaelklishin]

Your are welcome to investigate and propose a PR. This hasn't been a topic anyone has brought up since 2016-2017 (that I can remember).

The only alternative to the status quo I can think of would be a "force close connection" command that
would shut down rabbit_reader processes without going through a connection.close sequence, much like #abort does in clients.

[johanrhodin]

Yes, reproduced with heartbeats too. @gomoripeti and I'll discuss. I realize there are much bigger fish to fry, this is just an interesting edge-case.