[Bug]: when a private registry (url/credentials) is specified through System properties, public images cannot be pulled.

[StFS]

Module

Core

Testcontainers version

1.17.6

Using the latest Testcontainers version?

Yes

Host OS

MacOS / Linux

Host Arch

x86

Docker version

Client:
 Cloud integration: v1.0.29
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:18 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.14.1 (91661)
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:00:19 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

The problem happens for us when we run in our CI environment. The CI job that runs maven integration tests is run inside a fresh container that does not have any docker images cached. We run the maven job and specify a private registry and that seems to result in the public images that are needed (such as ryuk, mysql, kafka and others) not being fetched and erroring with an "invalid username/password" error.

I've managed to reproduce this locally (on a Mac) by following these steps:

• Make sure your local docker image cache is empty. Running docker images should return an empty list. You can get far at least (maybe even all the way) by running docker system prune -a
• Make sure that you remove any configuration of your docker setup (move your ~/.docker/config.json file somewhere else) so that you’re basically starting with a clean slate (same as you would be with a clean CI container).
• With a java project that uses testcontainers with images from a private registry that requires authentication, make sure that registry is not configured to allow passthough proxying of public images from the public docker registry.
• In that project run: mvn -Dregistry.url=foobar.myregistry.com -Dregistry.username=fluff -Dregistry.password=bunny verify (of course, using the correct url, username and password).
  • This run will fail because it won’t be able to get the public images such as ryuk. You will see a bunch of com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Get https://registry-1.docker.io/v2/testcontainers/ryuk/manifests/0.3.4: unauthorized: incorrect username or password"}
• Now, if you try running the maven command again, but not providing the custom registry url/user/pass: mvn verify
  • This run will also fail. The public images will be retrieved but the private ones will not be.
• Finally, if you run the command one more time, with the private registry url/user/pass, it works because the step above seems to have fetched the public images and stored it in the local docker cache.

Relevant log output

An excerpt from the log output when running the maven command which specifies the private registry:


# We use GitLab which offers a private registry, just a hint if you need to set one up quickly.
# mvn -Dregistry.url=registry.gitlab.com -Dregistry.username=gitlab-token -Dregistry.password=some_very_secure_password verify

...

39665 [main] WARN  🐳 [testcontainers/ryuk:0.3.4]  - Retrying pull for image: testcontainers/ryuk:0.3.4 (81s remaining)
40434 [docker-java-stream--876689195] ERROR com.github.dockerjava.api.async.ResultCallbackTemplate  - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Head \"https://registry-1.docker.io/v2/testcontainers/ryuk/manifests/0.3.4\": unauthorized: incorrect username or password"}

	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.base/java.lang.Thread.run(Thread.java:833)

Note that the error message may change to something like "toomanyrequests" at some point but the first errors should complain about incorrect username/password at least.

Additional Information

I asked about this on the Slack channel and got some good feedback from @kiview . A link to that thread is here: https://testcontainers.slack.com/archives/C1SUBPZK6/p1673350403349039

He points out that this may be something that needs to be fixed in

testcontainers-java/core/src/main/java/org/testcontainers/dockerclient/AuthDelegatingDockerClientConfig.java

Line 27 in de1324e

public AuthConfig effectiveAuthConfig(String imageName) {

. It should make sure that the private image registry authentication is only used when fetching images from that registry and when fetching public images it should not be using the auth credentials.

[kiview]

Thanks for sharing this detailed issue @StFS. Can you please configure org.testcontainers and com.github.dockerjava loggers to DEBUG level (see https://www.testcontainers.org/supported_docker_environment/logging_config/ for logging config) and share the logs of the failing run with us in this issue?

From looking at the code, I would expect that the fallbackAuthConfig is used, which would maybe unconditionally use the auth config configured through properties, irrespective of the image name:
https://github.com/docker-java/docker-java/blob/079797ffb278a48ebc5db6d1032f36f0f1ec7243/docker-java-core/src/main/java/com/github/dockerjava/core/DefaultDockerClientConfig.java#L284

We might consider this an upstream bug from docker-java we want to fix there, or we can fix it in our AuthDelegatingDockerClientConfig implementation downstream.

[StFS]

Thanks for sharing this detailed issue @StFS. Can you please configure org.testcontainers and com.github.dockerjava loggers to DEBUG level (see https://www.testcontainers.org/supported_docker_environment/logging_config/ for logging config) and share the logs of the failing run with us in this issue?

weeeeell... after looking at those logs I can see that they log out the authentication credentials in base64 encoding. So I'd be careful requesting people post those logs in public issues like this. I'm creating a temporary login to use for this issue which I will use to generate those logs and I'll share those in a bit.

We might consider this an upstream bug from docker-java we want to fix there, or we can fix it in our AuthDelegatingDockerClientConfig implementation downstream.

From the response that this issue received in docker-jave, I'm skeptical that they'll fix it. Although that makes a lot of sense to me because I think this behavior is kinda silly.

[kiview]

We are also maintainers of docker-java, so we can consider to fix it in the upstream if it is considered a bug in that context.

[StFS]

Attached are debug logs that I've modified a bit to remove as much of sensitive information as I could find.
Edit: Removed log

[kiview]

I think you forgot to set <logger name="com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.wire" level="OFF"/> as per documentation. That cleans up the logs considerably and also removes the output of sensitive information. I am sorry if this was not clear enough from the logs, we might consider to make this setting more explicitly.

Can you please share again with this config activated?

[StFS]

Sorry about that... my bad.

I've turned the wire log off as you instructed but I still see that the X-Registry-Auth header is still being logged out:

118730 [docker-java-stream--876689195] DEBUG com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.headers  - http-outgoing-0 >> X-Registry-Auth: ___REDACTED___

I can remove those lines manually... just something to be aware of.

[StFS]

Here is the log (with http.wire logging turned off), but as I said, I modified it a bit to remove the X-Registry-Auth header that was being logged out.

mvn-testcontainers-debug.2.log

[kiview]

I've turned the wire log off as you instructed but I still see that the X-Registry-Auth header is still being logged out:

Thanks a lot for sharing, we will adapt the docs to also recommend turning off .

Just for sake of documenting the potential workaround we discussed in Slack, I'll explain it here for users discovering this issue.

This issue seems to occur if docker-java's DockerClientConfig is configured through System properties. Testcontainers' AuthDelegatingDockerClientConfig seems to also not work around this issue. As a workaround (and maybe good solution for Testcontainers in general), users can add a ~/.docker/config.json file if the Docker CLI is not available and hence docker login can not be executed (e.g., on CI). The file has the following format:

{
    "auths": {
        "https://registry.example.com/": {
            "auth": "X2pzb25fa2V5OnsKInR5cGUiOiAic2VydmljZV9hY2NvdW50IiwKInByb2plY3RfaWQiOiAiZXhhbXBsZS1wcm9qZWN0IiwKInByaXZhdGVfa2V5X2lkIjogImV4YW1wbGUta2V5LWlkIiwKInByaXZhdGVfa2V5IjogImV4YW1wbGUtcHJpdmF0ZS1rZXkiLAoiY2xpZW50X2VtYWlsIjogInVzZXJAZXhhbXBsZS5jb20iLAoiY2xpZW50X2lkIjogInVzZXJAZXhhbXBsZS5jb20iLAoiYXV0aF91cmkiOiAiaHR0cHM6Ly9yZWdpc3RyeS5leGFtcGxlLmNvbS9vL29hdXRoMi9hdXRoIiwKInRva2VuX3VyaSI6ICJodHRwczovL3JlZ2lzdHJ5LmV4YW1wbGUuY29tL3Rva2VuIiwKImF1dGhfcHJvdmlkZXJfeDUwOV9jZXJ0X3VybCI6ICJodHRwczovL3JlZ2lzdHJ5LmV4YW1wbGUuY29tIiwKImNsaWVudF94NTA5X2NlcnRfdXJsIjogImh0dHBzOi8vcmVnaXN0cnkuZXhhbXBsZS5jb20iCn0="
        }
    }
}

with auth being:

echo -n 'username:password' | base64

In the future, #6238 should also provide a good alternative solution.

[kiview]

From the logs, these lines hint us at the issue, which corresponds to the assumptions I mentioned before:

1082 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator  - Looking up auth config for image: testcontainers/ryuk:latest at registry: https://index.docker.io/v1/
1083 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator  - RegistryAuthLocator has configFile: /Users/stefan.freyr/.docker/config.json (does not exist) and commandPathPrefix:
1083 [main] INFO  org.testcontainers.utility.RegistryAuthLocator  - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:latest, configFile: /Users/stefan.freyr/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /Users/stefan.freyr/.docker/config.json (No such file or directory)
1085 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator  - No matching Auth Configs - falling back to defaultAuthConfig [AuthConfig{username=gitlab-token, password=hidden non-blank value, auth=blank, email=null, registryAddress=registry.mycompany.com, registryToken=blank}]
1085 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig  - Effective auth config [AuthConfig{username=gitlab-token, password=hidden non-blank value, auth=blank, email=null, registryAddress=registry.mycompany.com, registryToken=blank}]

There is no explicit auth config accessible to Testcontainers configured, so Testcontainers falls back to the default docker-java's, fallbackAuthConfig, and it is used without taking the registry part of the image into account. We can definitely fix it in our Testcontainers implementation, if we ignore the fallbackAuthConfig if it is for a different registry.

But I think it is actually a bug in docker-java's DefaultDockerClientConfig that returns a System property configured auth config, even if it was configured for another registry. We have code in docker-java that resolves auth details for the registry within the image name, but it seems that by using the System property configuration, this code is circumvented.

[eddumelendez]

Hi, creating DOCKER_AUTH_CONFIG env var with json

{
    "auths": {
        "https://registry.example.com/": {
            "auth": "X2pzb25fa2V5OnsKInR5cGUiOiAic2VydmljZV9hY2NvdW50IiwKInByb2plY3RfaWQiOiAiZXhhbXBsZS1wcm9qZWN0IiwKInByaXZhdGVfa2V5X2lkIjogImV4YW1wbGUta2V5LWlkIiwKInByaXZhdGVfa2V5IjogImV4YW1wbGUtcHJpdmF0ZS1rZXkiLAoiY2xpZW50X2VtYWlsIjogInVzZXJAZXhhbXBsZS5jb20iLAoiY2xpZW50X2lkIjogInVzZXJAZXhhbXBsZS5jb20iLAoiYXV0aF91cmkiOiAiaHR0cHM6Ly9yZWdpc3RyeS5leGFtcGxlLmNvbS9vL29hdXRoMi9hdXRoIiwKInRva2VuX3VyaSI6ICJodHRwczovL3JlZ2lzdHJ5LmV4YW1wbGUuY29tL3Rva2VuIiwKImF1dGhfcHJvdmlkZXJfeDUwOV9jZXJ0X3VybCI6ICJodHRwczovL3JlZ2lzdHJ5LmV4YW1wbGUuY29tIiwKImNsaWVudF94NTA5X2NlcnRfdXJsIjogImh0dHBzOi8vcmVnaXN0cnkuZXhhbXBsZS5jb20iCn0="
        }
    }
}

should fix the issue.