[Bug]: MySQL Testcontainer can't chown /etc/mysql/conf.d folder inside rootless docker container

[jsloop42]

Module

MySQL

Testcontainers version

1.19.8

Using the latest Testcontainers version?

Yes

Host OS

Ubuntu 22.04

Host Arch

x86_64 Amazon Workspace

Docker version

Docker version 27.0.3, build 7d4bcd8

What happened?

Getting the following error when trying to run local test cases in Ubuntu. This is the same issue reported with Podman here.

com.example.db-utils-testExecution error (InternalServerErrorException) at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder/execute (DefaultInvocationBuilder.java:247).
Status 500: {"message":"failed to Lchown \"/etc/mysql/conf.d\" for UID 1467898976, GID 1467800513: lchown /etc/mysql/conf.d: invalid argument"}

This issue is happening for us with Ubuntu 22.04 with Clojure and kaocha where we are using testcontainer for running tests locally that spins up MySQL. It however works with Gitlab pipeline. This is really blocker for us. The same setup works on a mac.

I have tried overriding the MySQLContainer from Clojure side using proxy as mentioned in the linked bug, but it is not working. I don't get any error at this time, but the tests get stuck. I am also not able trace any logs because I couldn't find a way to enable printing logs in kaocha.

Kindly requesting your input on how to solve this. Thanks.

org.testcontainers/mysql {:mvn/version "1.19.8"}
org.testcontainers/localstack {:mvn/version "1.19.8"}
org.testcontainers/testcontainers {:mvn/version "1.19.8"}

Relevant log output

No response

Additional Information

No response

[kiview]

This issue is happening for us with Ubuntu 22.04 with Clojure and kaocha where we are using testcontainer for running tests locally that spins up MySQL. It however works with Gitlab pipeline.

Can you elaborate on what this means? Using these Testcontainers tests within Gitlab work? Or when running the container through Gitlab service integration?

The Podman issue you linked seems different to me, at least it reports a different error message. Can you please share the full stack trace? And this is running a rootless Docker daemon on your Ubuntu?

[jsloop42]

I was getting error message like this before.

...
Instrumented example.core.job
Instrumented 112 namespaces in 28.5 seconds.
--- unit (clojure.test) ---------------------------
com.example.quote.api-testExecution error (InternalServerErrorException) at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder/execute (DefaultInvocationBuilder.java:247).
Status 500: {"message":"failed to Lchown \"/etc/mysql/conf.d\" for UID 1467898976, GID 1467800513 (try increasing the number of subordinate IDs in /etc/subuid and /etc/subgid): lchown /etc/mysql/conf.d: invalid argument"}

But recent error messages are like the ones I posted in the main issue. It has less information.

Please ignore the Gitlab comment. The main issue is it works on macOS and not on Ubuntu which is part of AWS Workspace. I have not tested in an Ubuntu OS running on a physical computer, but I think the result might be the same.

The docker is running in rootless mode and this is the source of the issue. Because my colleague was able to change the UID and GID to the one given in the error message and added one of mysql and he is able to run all the tests without any issue. But his docker is running as root.

If I update the UID and GID then I am able to run the tests but my service fails to start as docker daemon is unable to start after the system restart. I think this is because of some conflict in the UID and GID for my user. Because when I ran docker rootless install setup, it printed a command which has the UID and GID as 100000:65536.

We are investigating this, but our last resort is to run everything as root.

[Clortox]

My team is having the exact same issue as you on our AWS workspaces. I have tried incrementing the final field in /etc/sub{gid,uid}, but that hasn't fixed it. Have you found any other solutions other than just running as root?

[jsloop42]

No. Currently we are running all services as root. Then run the test case as normal user if I remember correctly. The key here is to run docker as root and not in rootless mode which doesn't work.

[ventod]

I'm having the same problem. From my work pc, where I can't run anything as root.