[Bug]: PrefixingImageNameSubstitutor does not work if docker.io explicitly used as a registry

[frederikb]

Module

Core

Testcontainers version

1.18.3

Using the latest Testcontainers version?

Yes

Host OS

Linux and MacOS

Host Arch

x86-64 and ARM

Docker version

Client:
 Cloud integration: v1.0.33
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:51:16 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.20.1 (110738)
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:50:59 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

I'm using the default prefixing image substitutor introduced with #3413.

Activated via .testcontainers.properties with hub.image.name.prefix=my.custom.registry/dockerhubproxy/

The configuration is applied and I can see that the image substitution is getting processed in general. The issue is that the logic to detect whether or not to perform image name substitution is as follows:

boolean isAHubImage = original.getRegistry().isEmpty();

(see PrefixingImageNameSubstitutor.java )

This only catches the case whereas no image registry was included (default registry) and then presumes that we meant DockerHub and thus would like to perform Image Name Substitution. If we have a docker registr stated explicitely such as docker.io/postgres:14 then image name substitution is not applied.

If the goal is to always perform image name substitution for DockerHub images then I would count this as a bug.

The check should rather be something like:

boolean isAHubImage = original.getRegistry().isEmpty() || "docker.io".equals(original.getRegistry());

or better yet move that check into the DockerImageName class itself.

Workaround: make sure you do not include docker.io/ when configuring image names. Unfortunately this means that the defaults applied by Quarkus Dev Services do not work with prefixing image name substitution.

Relevant log output

2023-06-26 10:09:10,398 INFO  [tc.doc.io/postgres:14] (build-58) Pulling docker image: docker.io/postgres:14. Please be patient; this may take some time but only needs to be done once.
2023-06-26 10:09:10,402 INFO  [org.tes.uti.RegistryAuthLocator] (build-58) Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: docker.io/postgres:latest, configFile: /home/runner/.docker/config.json, configEnv: DOCKER_AUTH_CONFIG). Falling back to docker-java default behaviour. Exception message: Status 404: No config supplied. Checked in order: /home/runner/.docker/config.json (file not found), DOCKER_AUTH_CONFIG (not set)
2023-06-26 10:09:12,107 INFO  [tc.doc.io/postgres:14] (docker-java-stream--933788147) Starting to pull image
2023-06-26 10:09:12,227 INFO  [tc.doc.io/postgres:14] (docker-java-stream--933788147) Pulling image layers:  0 pending,  0 downloaded,  0 extracted, (0 bytes/0 bytes)


...


2023-06-26 10:09:21,122 INFO  [org.tes.uti.ImageNameSubstitutor] (build-58) Using my.custom.registry/dockerhubproxy/testcontainers/ryuk:0.4.0 as a substitute image for testcontainers/ryuk:0.4.0 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
2023-06-26 10:09:21,124 INFO  [tc.my.cus.reg.4.0] (build-58) Pulling docker image: my.custom.registry/dockerhubproxy/testcontainers/ryuk:0.4.0. Please be patient; this may take some time but only needs to be done once.
2023-06-26 10:09:21,124 INFO  [org.tes.uti.RegistryAuthLocator] (build-58) Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: my.custom.registry/dockerhubproxy/testcontainers/ryuk:latest, configFile: /home/runner/.docker/config.json, configEnv: DOCKER_AUTH_CONFIG). Falling back to docker-java default behaviour. Exception message: Status 404: No config supplied. Checked in order: /home/runner/.docker/config.json (file not found), DOCKER_AUTH_CONFIG (not set)
2023-06-26 10:09:23,870 INFO  [tc.my.cus.reg.4.0] (docker-java-stream--1767320720) Starting to pull image
2023-06-26 10:09:23,871 INFO  [tc.my.cus.reg.4.0] (docker-java-stream--1767320720) Pulling image layers:  0 pending,  0 downloaded,  0 extracted, (0 bytes/0 bytes)

Additional Information

No response

[frederikb]

I noticed that the PrefixingImageNameSubstitutorTest contains the test cases hubIoRegistryIsNotChanged and hubComRegistryIsNotChanged. It seems therefore as if this behavior was intended?

I guess changing is behavior would therefore constitute a breaking change for those using this ImageNameSubstitutor. What would you suggest, go ahead with the change? Hide it behind a configuation property or stablish a separate ImageNameSubstitutor?

[SirSeven]

Hi @frederikb did you find a easy solution for this?

We are running various quarkus projects with TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX set to a local pull-through docker registry. But the Quarkus project is prefixing all their database containers with docker.io/ (see https://github.com/quarkusio/quarkus/blob/26db59d0b672053df63fd5c058b0778ffaac4739/build-parent/pom.xml#L86), which prevents the PrefixingImageNameSubstitutor from adding the local registry prefix. I've built my own PrefixingImageNameSubstitutor that would replace docker.io, but adding it to hundrets of projects makes it a pain.

It could be as easy as another env variable stating "replace docker-hub prefixes too":

        boolean isAHubImage = original.getRegistry().isEmpty();
        if (!isAHubImage && replaceDockerHubPrefix) {
           isAHubImage = original.getRegistry().equalsIgnoreCase("docker.io");
        }
        if (!isAHubImage) {
            log.debug("Image {} is not a Docker Hub image - not applying registry/repository change", original);
            return original;
        }

        log.debug("Applying changes to image name {}: applying prefix '{}'", original, configuredPrefix);

        DockerImageName prefixAsImage = DockerImageName.parse(configuredPrefix);

        return original
                .withRegistry(prefixAsImage.getRegistry())
                .withRepository(prefixAsImage.getRepository() + original.getRepository());

[frederikb]

Hi @SirSeven,

we kept using the mentioned workaround which means that we explicitely stated the image name for each dev service in order to force the use of the configured proxy:

i.e. quarkus.datasource.devservices.image-name=postgres:16 (without the docker.io prefix) which additionally has the benefit of making sure that it matches the major version of the database / message queue used in production.