Docker pull fails when run from a cronjob

[chrisforrette]

Hello there, thank you for building this helper!

I'm having an issue where we've configured this ECR helper on an EC2 instance and it works fine to docker pull an image during an SSH session, but I wrote a cron job to periodically pull a specific image and it fails like so:

Sep 13 23:47:01 localhost CRON[9224]: (root) CMD (docker pull XXXXXX.dkr.ecr.us-west-1.amazonaws.com/my-image-repo:latest)
Sep 13 23:47:01 localhost dockerd[1298]: time="2024-09-13T23:47:01.737021888Z" level=info msg="Attempting next endpoint for pull after error: Get https://XXXXXX.dkr.ecr.us-west-1.amazonaws.com/v2/my-image-repo/manifests/latest: no basic auth credentials"
Sep 13 23:47:01 localhost dockerd[1298]: time="2024-09-13T23:47:01.737075289Z" level=error msg="Handler for POST /v1.40/images/create returned error: Get https://XXXXXX.dkr.ecr.us-west-1.amazonaws.com/v2/my-image-repo/manifests/latest: no basic auth credentials"

My /root/.docker/config.json looks like this:

{
	"auths": {
		"XXXXXX.dkr.ecr.us-west-1.amazonaws.com": {}
	},
	"credsStore": "ecr-login"
}

And I see nothing appending to ~/.ecr/log/ecr-login.log when these run.

My suspicion is that perhaps ~/.docker/config.json is not loaded during cron jobs but I can't find much documentation about it. Do you happen to know if this is the case? Or if there might be something else going on?

Thanks!

[sondavidb]

Hey, thanks for bringing this up!

"XXXXXX.dkr.ecr.us-west-1.amazonaws.com": {}

Is everything besides the XXXXXX exactly how it is in the config, or did you just censor it? If it's the former, this might explain the behavior. My guess is that it went to the auths section of your config, saw the lack of any authentication in there, and returned an error. If you remove the auths section I think it should work. Generally speaking, since auths and credsStore perform the same duties, it's best to only use one or the other to avoid any confusion.

[chrisforrette]

@sondavidb Hi there, thanks for the response!

Apologies, I should've made it clearer but yeah the XXXXXX is just a redacted AWS account number but everything else is the same.

I made the change you suggested—removing the auths key from ~/.docker/config.json—but it doesn't seem to have changed anything.

[sondavidb]

Thanks. I attempted to reproduce this with a cron job that runs once every minute but couldn't get it to reproduce. Can you make sure this is similar enough to your setup?

Commands

dummyuser@my-ip:~$ crontab -l
[redacted as useless comment]
* * * * * /home/dummyuser/a.sh > /home/dummyuser/cron.out
dummyuser@my-ip:~$ cat a.sh
docker pull 123456789012.dkr.ecr.us-west-2.amazonaws.com/dummyrepo:alpine
dummyuser@my-ip:~$ cat ~/.docker/config.json
{
        "auths": {
                "123456789012.dkr.ecr.us-west-2.amazonaws.com": {
                }
        },
        "credsStore": "ecr-login"
}
dummyuser@my-ip:~$ image ls ^C
# Wait for cron job to run
dummyuser@my-ip:~$ docker image ls
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
dummyuser@my-ip:~$ docker image ls
REPOSITORY                                               TAG       IMAGE ID       CREATED         SIZE
123456789012.dkr.ecr.us-west-2.amazonaws.com/dummyrepo   alpine    8ca4688f4f35   11 months ago   7.34MB
dummyuser@my-ip:~$ cat cron.out
alpine: Pulling from dummyrepo
96526aa774ef: Pulling fs layer
96526aa774ef: Verifying Checksum
96526aa774ef: Download complete
96526aa774ef: Pull complete
Digest: sha256:97e4616341f8109256b81903bd2be6ceb302c10065ef9f7de88a6fb5e8222680
Status: Downloaded newer image for 123456789012.dkr.ecr.us-west-2.amazonaws.com/dummyrepo:alpine
123456789012.dkr.ecr.us-west-2.amazonaws.com/dummyrepo:alpine

FWIW, it's erroring out for me if running the job with sudo, but if I copy the config to /root/.docker/config.json like what it seems like you're doing, it works just fine.

[sondavidb]

Hey @chrisforrette just wanted to check in again and see if the setup above looks correct