Feature request: support `docker.io/*` images to use auth config from `~/.docker/config.json`

[achaphiv]

Version: org.testcontainers:testcontainers-bom:1.16.0

This works fine:

new GenericContainer<>("amazon/dynamodb-local:1.16.0")

I've been seeing more recommendations to prefer fully qualified docker images (e.g. podman), so I switched to:

new GenericContainer<>("docker.io/amazon/dynamodb-local:1.16.0")

And then errors started occurring during my CI build.

I eventually figured out that the auth config was not being picked up:

[Test worker] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: docker.io/amazon/dynamodb-local:1.16.0
[Test worker] DEBUG org.testcontainers.images.AbstractImagePullPolicy - Using locally available and not pulling image: docker.io/amazon/dynamodb-local:1.16.0
[Test worker] INFO 🐳 [docker.io/amazon/dynamodb-local:1.16.0] - Creating container for image: docker.io/amazon/dynamodb-local:1.16.0
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: docker.io/amazon/dynamodb-local:1.16.0 at registry: docker.io
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /home/builder/.docker/config.json (exists) and commandPathPrefix: 
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [docker.io] for dockerImageName [docker.io/amazon/dynamodb-local:1.16.0]
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]

So I switched to index.docker.io, and it worked.

new GenericContainer<>("index.docker.io/amazon/dynamodb-local:1.16.0"):

[Test worker] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: index.docker.io/amazon/dynamodb-local:1.16.0
[Test worker] DEBUG org.testcontainers.images.AbstractImagePullPolicy - Using locally available and not pulling image: index.docker.io/amazon/dynamodb-local:1.16.0
[Test worker] INFO 🐳 [index.docker.io/amazon/dynamodb-local:1.16.0] - Creating container for image: index.docker.io/amazon/dynamodb-local:1.16.0
[Test worker] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Delegate call to effectiveAuthConfig failed with cause: 'Invalid repository name, try "amazon/dynamodb-local" instead'. Resolution of auth config will continue using RegistryAuthLocator.
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: index.docker.io/amazon/dynamodb-local:1.16.0 at registry: index.docker.io
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - Cached auth found: [AuthConfig{username=redacted, password=hidden non-blank value, auth=hidden non-blank value, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]

However, this is out of line with the docker cli, which does read from the ~/.docker/config.json.

If I change the https://index.docker.io/v1/ auth value and try to pull via:

docker -D pull docker.io/amazon/dynamodb-local:1.16.0

It fails with:

Error response from daemon: Head "https://registry-1.docker.io/v2/amazon/dynamodb-local/manifests/1.16.0": unauthorized: incorrect username or password

[rnorth]

Ah interesting - it sounds like there might be some magic behaviour in Docker CLI, mapping docker.io to another hostname. If this is the case, we'd have to emulate that, potentially.

Are you using hardcoded credentials in your ~/.docker/config.json file, or are you using a creds helper/store? (Looking in the file it should be fairly obvious which is in effect).

[achaphiv]

My CI runs:

docker login -u ${DOCKERHUB_USERNAME} -p ${DOCKERHUB_PASSWORD}
# ...
./gradlew clean build

[rnorth]

Depending on what's already in the config file, that docker login could be either writing hardcoded creds or pushing the creds into an helper/store.

[achaphiv]

This is AWS codepipeline/codebuild, so it just starts fresh each time.

I checked via cat ~/.docker/config.json, and it's just hardcoded { auths: { ... } }. No credentials store.

[sschulze]

In my case the docker.io registry is hardcoded in a library (specifically it's the reference to docker.io/vectorized/redpanda in the quarkus-kafka devservice).
A rather dirty hack is to duplicate the auth entry in the ~/.docker/config.json to the name "docker.io". Sadly I have to do this manually with jq because docker login docker.io just creates the entry for the fully qualified hostname.
This ist the line I use:

jq '.auths["docker.io"] = .auths[.auths|keys[]|select(.|test("http.*docker.io"))]' ~/.docker/config.json > ~/.docker/config.json.modified && \
mv ~/.docker/config.json.modified ~/.docker/config.json

[GUSTAVPEREIRA]

I got a similar error but i don't know if is necessary another issue. But for some reason I can't able to create any.
But I got this error using java 17 and version 1.21.3 from maven and quarkus.

org.testcontainers
testcontainers-bom

When I tried run any test that depends from redpanda I got this error:

Runtime java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[error]: Build step io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor#startKafkaDevService threw an exception: java.lang.RuntimeException: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=docker.io/vectorized/redpanda:v22.3.4, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@b68813f)
at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.startKafkaDevService(DevServicesKafkaProcessor.java:105)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:858)
at io.quarkus.builder.BuildContext.run(BuildContext.java:282)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at java.base/java.lang.Thread.run(Thread.java:840)
at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=docker.io/vectorized/redpanda:v22.3.4, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@b68813f)

If this deserve another issue please, I'm glad if any could create one for me.