Authorization failure downloading docker image from gcr.io due to 'docker-credential-gcloud' not found

[aluckenbaugh]

Problem

If TestContainers detects that an image uploaded to a protected gcr.io docker registry needs to be downloaded it will fail when trying to authenticate using the Google Cloud SDK when running on Windows. At the root it seems this is caused by the org.testcontainers.utility.RegistryAuthLocator#getCredentialProgramName() function calculating "docker-credential-gcloud" for the command line to execute, but the actual filename from the google-cloud-sdk is "docker-credential-gcloud.cmd". Therefore an exception like the following is logged and the image pull fails.

Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: gcr.io/path/to/myimage:latest, configFile: C:\Users\myusername.docker\config.json. Falling back to docker-java default behaviour. Exception message: Could not execute [docker-credential-gcloud, get]. Error=2, The system cannot find the file specified

Please note the directory containing "docker-credential-gcloud.cmd" is indeed on the system PATH.

Logs

10:49:23.379 INFO [🐳 ] ( agroal-11) Pulling docker image: gcr.io/path/to/myimage:latest. Please be patient; this may take some time but only needs to be done once.
10:49:23.380 DEBUG [ or.te.ut.RegistryAuthLocator] ( agroal-11) Looking up auth config for image: gcr.io/path/to/myimage:latest at registry: gcr.io
10:49:23.380 DEBUG [ or.te.ut.RegistryAuthLocator] ( agroal-11) RegistryAuthLocator has configFile: C:\Users\myusername.docker\config.json (exists) and commandPathPrefix:
10:49:23.381 DEBUG [ or.te.ut.RegistryAuthLocator] ( agroal-11) registryName [gcr.io] for dockerImageName [gcr.io/path/to/myimage:latest]
10:49:23.381 DEBUG [ or.te.ut.RegistryAuthLocator] ( agroal-11) Executing docker credential provider: docker-credential-gcloud to locate auth config for: gcr.io
10:49:23.381 DEBUG [ or.te.sh.or.ze.ex.ProcessExecutor] ( agroal-11) Executing [docker-credential-gcloud, get].
10:49:23.383 DEBUG [ or.te.ut.RegistryAuthLocator] ( agroal-11) Failure running docker credential helper/store (docker-credential-gcloud)
10:49:23.383 INFO [ or.te.ut.RegistryAuthLocator] ( agroal-11) Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: gcr.io/path/to/myimage:latest, configFile: C:\Users\myusername.docker\config.json. Falling back to docker-java default behaviour. Exception message: Could not execute [docker-credential-gcloud, get]. Error=2, The system cannot find the file specified
10:49:23.383 DEBUG [ or.te.ut.RegistryAuthLocator] ( agroal-11) No matching Auth Configs - falling back to defaultAuthConfig [null]
10:49:23.383 DEBUG [or.te.do.AuthDelegatingDockerClientConfi] ( agroal-11) Effective auth config [null]
10:49:24.251 ERROR [ co.gi.do.ap.as.ResultCallbackTemplate] (docker-java-stream--12250) Error during callback: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"}

Environment details:

• Windows 10
• gcloud version:
  Google Cloud SDK 340.0.0
  bq 2.0.67
  core 2021.05.07
  gsutil 4.61
• TestContainers version: 1.15.3
• Docker Desktop version 3.3.1
• Docker version: 20.10.5

Possible Solutions

1. Update the "gcloud" values in the %USERPROFILE%\.docker\config.json to be "gcloud.cmd"
   This json file is produced by invoking gcloud auth configure-docker and produces a json file like the following:

{
	"credHelpers": {
		"gcr.io": "gcloud",
		"us.gcr.io": "gcloud",
		"eu.gcr.io": "gcloud",
		"asia.gcr.io": "gcloud",
		"staging-k8s.gcr.io": "gcloud",
		"marketplace.gcr.io": "gcloud"
	},
	"credsStore": "desktop",
	"stackOrchestrator": "swarm"
}

This approach does not seem desirable though since it is a manual change and a regular docker pull for this container works fine.

2. Update RegistryAuthLocator to add the ".cmd" suffix to the command. I'm not sure if this is the best way to fix this issue, but I hacked this together quickly to test it out and it gets us past the problem for the moment. It requires a new member variable:

private static final boolean IS_OS_WINDOWS = System.getProperty("os.name").startsWith("Windows");

and an update to the protected constructor to initialize this.commandExtension when the OS is windows:

protected RegistryAuthLocator() {
    final String dockerConfigLocation = System.getenv().getOrDefault("DOCKER_CONFIG",
        System.getProperty("user.home") + "/.docker");
    this.configFile = new File(dockerConfigLocation + "/config.json");
    this.commandPathPrefix = "";

    if(IS_OS_WINDOWS) {
        // Generate the proper command extensions for cmd scripts.
        this.commandExtension = ".cmd";
    } else {
        this.commandExtension = "";
    }

    this.CREDENTIALS_HELPERS_NOT_FOUND_MESSAGE_CACHE = new HashMap<>();
}

[rnorth]

Thanks for the extensive analysis, and sorry you're having this problem.

I assume - but please verify - that a regular docker pull from a WSL2 or Cygwin shell works?

If that's the case, then honestly I think the second approach is reasonable. In this area, Testcontainers has to mimic the behaviour of Docker's CLI and how it interacts with the credential helpers/stores.

If Docker CLI works then I'd imagine, but haven't checked, that it's automatically adding .cmd to the command (or maybe trying both with/without .cmd). We should investigate that and implement the same behaviour in Testcontainers.

[jglink]

A regular docker pull does work and is currently needed for us to download an image from gcr.io

Only after changing the RegistryAuthLocator as shown above the download works directly with testcontainer.

[aluckenbaugh]

Yes, as @jglink wrote, a regular docker pull works for us from the command line with WSL2.

[rnorth]

I think we'll probably need to make Teatcontainers try with a .bat extension. I did check and Docker CLI doesn't have this workaround, which is a little strange.

When you're encountering this problem, are you launching Testcontainers tests in an IDE, perhaps? Is the same true when running tests under a WSL2 shell? This might explain why the extension is necessary, and also why the Docker CLI itself doesn't seem to need it.

[aluckenbaugh]

Sorry, I think I misspoke earlier: we have been running TestContainers in a Maven project via both IntelliJ and a Windows Terminal / Command Prompt session and have observed the same errors in each. As far as I know our developers haven't been running when inside a WSL2 shell specifically though.

At least with these versions:
Google Cloud SDK 340.0.0
bq 2.0.67
core 2021.05.07
gsutil 4.61

the docker-credential-gcloud.cmd script doesn't have a corresponding *.bat file, so it seems like it would be necessary to use a .cmd extension.

[rnorth]

Thanks for clarifying - also thanks for correcting me about the extension.

[CoryHagerman]

I think this issue has been fixed via code in master.

4aef1dc

I am not sure it was done to directly fix this bug but building and running master with this change has resolved the above issue for me.

Is there a timeline of when new releases are made from master for when this may be available in artifacts?

[eddumelendez]

Closing as resolved. Let us know if it is not.