"memory resource limit alarm" causes communication to PLCs to be slow or completely stop

[pfeifi01]

Describe the bug

We have noticed a slowdown in communication on our PLCs and at some point we had a complete communication loss. We have checked the rabbitmq log file and found some warnings and errors. It seems that there are some issues related to writing to the log-file itself.

The full log file can be downloaded here, the file size was too big for github.

Here is a screenshot from the RabbitMQ Web Interface (Overview)

We have noticed, that in the log file, which you can see in the screenshot, that there are logs from 12:54 written between logs from 12:53, so maybe this is an indicator for you to understand what's going wrong here.

We have noticed also this error line, which came ~100 times in a row:

2023-08-28 12:51:46.295000+02:00 [error] <0.2557.1> writing to MQTT socket #Port<0.1834> failed: badarg_

Reproduction steps

???

Expected behavior

no communication interrupts to PLC

Additional context

No response

[mkuratczyk]

A memory alarm is a deliberate mechanism, so the fact that it was triggered is not a bug: https://www.rabbitmq.com/alarms.html

You node status screenshot doesn't match that situation - only 233MB of memory was used at the time you took the screenshot.

Judging from the logs, it seems like it's 3.12, but please provide the exact version.

[pfeifi01]

The screenshot was taken at the time where I did the issue, the thing is just, that the slow communication still is present in the system.

yes, the version is 3.12.0

[michaelklishin]

What is a "PLC"?

A triggered memory alarm simply indicates that publishers
have been outpacing consumers for some time, possibly because there were no consumers. Message TTL and max queue length limits are two protection mechanisms
that exist for environments where this may be the case, and in particular those where MQTT clients
use QoS 0 (that is, transient messages).

According to the log, your clients do use QoS 0 publishing, therefore you either run the risk of hitting the above resource alarms, or must use message TTL or better yet, a length limit for MQTT subscriber queues.

[error] <0.2557.1> writing to MQTT socket #Port<0.1834> failed

means that a socket write by RabbitMQ on an MQTT client connection has failed. If you have a lot of those, it means that something is up with networking on your node, in a way that affects MQTT
client connections (or possibly all activity of a node).

[michaelklishin]

At the bottom of the log you can find

2023-08-28 12:56:17.946000+02:00 [error] <0.694.0> Ranch listener {acceptor,{0,0,0,0},1883} had connection process started with rabbit_mqtt_reader:start_link/3 at <0.2557.1> exit with reason: einval

which is a yet another example of a socket failure (EINVAL) on an inbound TCP connection.

[pfeifi01]

Thanks for pointing the TCP connection issue out.

To answer your question, a PLC is a "programmable logic controller", see here for further info. We have multiple machines with PLCs that are controlling their behavior and using the RabbitMQ message broker to exchange information with them.

[ansd]

Starting at 12:51:39, the log file shows many times

writing to MQTT socket #Port<...> failed: badarg

both for the same and for different MQTT connection processes.

Only later at 12:54:12, the memory alarm kicks in.

I suggest instead of logging and then ignoring the send failure, we should log and then immediately close the MQTT connection as done in #8553 and #9271.
This will hopefully lead to less follow up errors such as truncated logs and memory alarms.

It seems that there are some issues related to writing to the log-file itself.

Yes, to prevent overload, the logger will drop entries if too many lines are logged at once indicated by drop mode in the logs.

[ansd]

Closing as #9271 has been merged. Please re-open if you still experience this issue in RabbitMQ >= 3.12.5

[pfeifi01]

Would it be possible to generate a 3.12.5 release or higher, which is not a beta version, because we'd like not to update the productive system with a beta version. The highest version/tag i could find below 3.13-beta was 3.12.4.

Thanks in advance

[michaelklishin]

@pfeifi01 3.12.5 will ship soon.