RabbitMQ docker container started spewing out comma separated numbers to stdout

[RyanWoodSilac]

We are running RabbitMQ in a Docker Swarm as a service with one container connecting to a django instance to handle scheduled tasks. The rabbitmq service has a mounted cifs volume pointing to a SAN share and has been working in this config for a couple of years. This weekend while it was processing daily tasks the container hit max CPU and memory and standard output started outputting comma separated numbers followed by \n in hundreds of thousands of rows. Here is an example:

117,112,86,11,103,114,111,117,112,95,105,110,100,\n 101,120,86,7,114,101,116,114,105,101,115,73,0,0,0,\n 0,9,116,105,109,101,108,105,109,105,116,65,0,0,0,\n 2,86,86,7,114,111,111,116,95,105,100,83,0,0,0,36,\n

There were also some text in the logs periodically:
del,no_ack},\n {{true,\n

We restarted the docker service a few times with the container behaving the same. We ended up pinning the container to one of the Docker nodes and mounting a local, empty volume for it to use and it started working as normal.

There were no indications of network interruption or other infrastructure issue associated with the swarm or the SAN volume. My first guess would be these are some sort of metrics or debug logs? What could cause rabbitmq to behave like this? I believe it is tied to the previous cifs volume but am not sure where to start. Having the rabbitmq server in this config is not ideal so we would like to go back to using the cifs storage so the service can move with the swarm for HA. However we are reluctant to do so and have the same issue happen again.

I am a Sysadmin more than a developer so go easy on the new guy :)

[lukebakken]

I have never seen behavior like this. Let's start with the basics...

• What version of RabbitMQ and Erlang?
• Are you using a cluster?
• You mention "CIFS" which makes me think you're using Windows network shares. Is that true?
• What does "the service can move with the swarm for HA" mean?

[RyanWoodSilac]

1. 3.11.9 and Erlang/OTP 25 [erts-13.1.5] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]
2. No
3. Yes, the network share is hosted by a Windows 2019 file server cluster using tfs
4. For HA, the container can be moved from docker node to docker node. With local storage, we would lose the state of RabbitMQ if the container was moved to the other node. By keeping the mnesia data on a share and mounting it to the container, we can move the container around if a node goes down or for resource balancing without losing the data.

[lukebakken]

PS with enough context from your logs we might be able to make sense of those numbers:

> erl
Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit:ns]

Eshell V14.0.2 (press Ctrl+G to abort, type help(). for help)
1> S = [117,112,86,11,103,114,111,117,112,95,105,110,100,101,120,86,7,114,101,116,114,105,101,115,73,0,0,0,0,9,116,105,109,101,108,105,109,105,116,65,0,0,0,2,86,86,7,114,111,111,116,95,105,100,83,0,0,0,36].
[117,112,86,11,103,114,111,117,112,95,105,110,100,101,120,
 86,7,114,101,116,114,105,101,115,73,0,0,0,0|...]

2> io:format("~s~n",[S]).
upV^+group_indexV^'retriesI^ ^ ^ ^      timelimitA^ ^ ^ ^"VV^'root_idS^ ^ ^ $

[RyanWoodSilac]

After having looked into this further our running theory is that this is storage related. The .IDX files in the queue for the vhost have a last write time to just before we saw the service start outputting the numbers. The log lines just prior to that state that the vhost had crashed and was not restarted. So without a volume to write messages to, it started dumping them to standard out. It looks like the numbers can be converted back to the data by converting them to ASC II characters.

If the vhost crashes and is not restarted, is this the normal behavior?

[lukebakken]

If the vhost crashes and is not restarted, is this the normal behavior?

Nope, like I said I've never seen this behavior. The output is probably from crashed Erlang processes dumping their state to the console.

[lukebakken]

@RyanWoodSilac some suggestions:

• Do not run RabbitMQ on docker on Windows. RabbitMQ is natively supported on Windows.
• Do not use a network share for RabbitMQ data, unless your message rates are so low that performance is not a concern. Even then I wouldn't trust it to reliably store RabbitMQ data.

[RyanWoodSilac]

RabbitMQ is running as a container on a RHEL 8 host. So I think we are ok there.
Is there another way to keep the mnesia data accessible if the container moves off the host short of using a share?