RabbitMQ 3.12.3 Fails to Start Due to Incorrect erlang.cookie Permissions in Docker Environment

[alfonso-ramirezTGCS]

Describe the bug

I am running the RabbitMQ 3.12.3 Docker image in a Linux environment. The RabbitMQ image operates alongside other Docker images, one of which uses RabbitMQ as its message broker.

The issue arose when the Docker containers were restarted: RabbitMQ failed to start correctly because the erlang.cookie had the wrong permissions. I do not map the erlang.cookie to any folder, nor do I pass a custom value to it, as I do not use multiple nodes.

I have not been able to replicate this issue again naturally. The only way I managed to recreate it was by connecting to the Docker container through a shell, changing the permissions of /var/lib/rabbitmq/.erlang.cookie to something other than 400 (e.g., 100), and then restarting the RabbitMQ container. This forced RabbitMQ to fail because the erlang.cookie did not have the correct permissions. However, in my case, the erlang.cookie was not modified by any process, not Docker, nor any other container, and it was not mapped to a specific folder on the host machine.

Restarting RabbitMQ after it failed to start due to the permission issue resolved the problem because the cookie was regenerated. However, I want to prevent this issue from occurring again. While I was able to notice and fix the issue quickly, I would prefer it not to happen in the first place.

I will attach the log, rabbitmq.conf, and the list of plugins I am using at the end of this issue.

1. Is it possible for RabbitMQ to automatically change the permissions of the erlang.cookie to the correct settings if it encounters the 'Error when reading .erlang.cookie /var/lib/rabbitmq/: eacces' error?
2. What implications could arise if this automatic correction is implemented?
3. Would this be considered as working as designed, with the only solution being to restart RabbitMQ so that the erlang.cookie is created again with the correct permissions?

Reproduction steps

Steps to Recreate (Forceful Recreation):

1. Start RabbitMQ Docker Container:

   docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.12.3

2. Connect to the Running RabbitMQ Container:

   docker exec -it rabbitmq /bin/bash

3. Change the Permissions of the erlang.cookie File:

   chmod 100 /var/lib/rabbitmq/.erlang.cookie

4. Restart the RabbitMQ Container:

   docker restart rabbitmq

5. Observe RabbitMQ Failing to Start:

   • Check the logs to confirm that RabbitMQ failed to start due to incorrect erlang.cookie permissions:

     docker logs rabbitmq

Expected behavior

• RabbitMQ should start correctly without issues related to erlang.cookie permissions.
• If the erlang.cookie permissions are incorrect, RabbitMQ should automatically correct them to the appropriate settings (400 or 600) instead of failing to start.

Additional context

Rabbitmq.conf:

#RABBITMQ CONFIGURATION FILE
listeners.tcp.default = 5672
management.tcp.port = 15672

#Authentication Properties
auth_mechanisms.1 = PLAIN

default_user = el_test
default_pass = el_test

log.console.formatter.single_line = on
log.connection.level = debug
log.channel.level = debug
log.queue.level = debug
log.mirroring.level = debug
log.federation.level = debug
log.upgrade.level = debug
log.default.level = debug
log.file.level = debug
log.console.level = debug
log.console = true

plugins:
[rabbitmq_shovel,rabbitmq_shovel_management,rabbitmq_management,rabbitmq_auth_mechanism_ssl]

Log:

tgcp_mbroker_log.log

[lukebakken]

in my case, the erlang.cookie was not modified by any process, not Docker, nor any other container, and it was not mapped to a specific folder on the host machine

I doubt that. Without a way to reproduce this issue, the simplest explanation is that something happened in your environment to change the file permissions.

In addition, please see Team RabbitMQ's community support policy. If you wish to receive free support from Team RabbitMQ, you MUST follow those guidelines.

[lukebakken]

To answer your questions:

Is it possible for RabbitMQ to automatically change the permissions of the erlang.cookie to the correct settings if it encounters the 'Error when reading .erlang.cookie /var/lib/rabbitmq/: eacces' error?

No, this is not possible.

Would this be considered as working as designed, with the only solution being to restart RabbitMQ so that the erlang.cookie is created again with the correct permissions?

Yes, this is how the Erlang VM works.

[alfonso-ramirezTGCS]

Thanks for the feedback. It's been quite tricky to recreate the issue, but if it happens again, I'll be ready to get the logs. If the issue was in my environment, I'm quite sure it was caused by something beyond my control, as I only map the /var/lib/rabbitmq/mnesia directory to my host machine, and no service or process has access or ownership of the /var/lib/rabbitmq:

VOLUME_LIST="-v ${EXT_DEPLOY}/enabled_plugins:/etc/rabbitmq/enabled_plugins"
VOLUME_LIST="${VOLUME_LIST} -v /cdrive/ext/persist:/var/lib/rabbitmq/mnesia"

I double-checked the Docker logs and Linux processes running at the time, and none of them can modify files in /var/lib/rabbitmq of the Docker container.

Given the difficulty in recreating this issue and your responses, it was likely a one-time occurrence that altered the file somehow. Just to double check with you, the Erlang VM could not have been the culprit in here when the docker container got restarted? That is, it modified the permissions when it got shutdown and then started again

[lukebakken]

Just to double check with you, the Erlang VM could not have been the culprit in here when the docker container got restarted? That is, it modified the permissions when it got shutdown and then started again

Think about all of the millions of Erlang VM instances running around the world, right now. RabbitMQ servers, Ericsson telephone switches, etc. All of them depend on the cookie file to have correct permissions to start up. If this were even an occasional issue due to a bug in the VM, it would be reported daily if not hourly.

[michaelklishin]

Is it possible for RabbitMQ to automatically change the permissions of the erlang.cookie to the correct settings

This would be a catastrophically bad idea, for a piece of software to tamper with its own shared secrets.

RabbitMQ 3.12 is out of community support => locking.