Docker compose permission error

[Taniya-Das]

Observed permission error
For ex. "PermissionError: [Errno 13] Permission denied: '/logstash/config/config/logstash.yml'", in es_logstash_setup container.

Reproduction

Not completely sure how to reproduce, this seems to depend on how docker has been installed. On Jos' computer, all is fine. On Taniya, it isn't, neither is it on Rinjani or Agung servers.
To reproduce: clone the project into a new directory and run docker compose up -d.

Actual

Permission denied errors on some services that need to write information to the data folder, or to the logstash folder. Some services, such as mysql, are fine.

Expected

No permission denied exceptions.

Manually circumventing the problem

You can circumvent the problem by manually changing the owners of the directories (this is done on rinjani and agung) or by manually changing the permissions on the directories. This is no real solution of course - it should just work out of the box!

Additional info

We tried to fix this by making sure the correct directories exist with the correct user (that's why we committed data/connectors/.gitkeep for instance). This makes sure that the directories are created by the same user that also cloned the repo - which doesn't have to be the user that docker expects...

[PGijsbers]

When running the elasticsearch container, I also had the same problem, which was fixed by making the data/elasticsearch folder owned by root:root (sudo chown -R root:root data/elasticsearch).

Then I encountered the permission error above, where changing ownership did not work (it seems that es_logstash_setup uses a different user). I had to give write permission to others: find logstash/ -type f -exec chmod 666 {} +

edit: it looks like es_logstash_setup is running as apprunner user.

[PGijsbers]

As es_logstash_setup is a short-lived service, I recommend we just run the container as root by adding user: root to the docker-compose.yaml for that container. Unless anyone is aware of an easy way to set up permissions for a docker user that doesn't interfere too much with access on the host machine. Alternatively, we could look at which files exactly the container is trying to access and more narrowly assign permissions.

[rishavtarway]

Hi @fkiraly @PGijsbers ,
I reviewed #224 and reproduced the context from the thread: es_logstash_setup can fail with permission errors when writing generated files (for example logstash.yml), depending on host/docker user mapping. The issue appears to be in the short-lived setup container permissions rather than application logic.

My plan is:

Reproduce from a clean docker compose up flow.
Apply the minimal fix by running es_logstash_setup with an explicit user in docker-compose.yaml (as suggested in the issue discussion), avoiding broad host chmod/chown workarounds.
Validate startup path and update docs with the rationale and expected behavior.

I’ll keep scope limited to this permission bug and open a small draft PR early for review.
Would you be okay with me working on #224 with this approach?

[rishavtarway]

I opened a scoped Draft PR with the minimal compose-level fix for es_logstash_setup permissions: #713. Scope is limited to setup-container permission behavior only.