Issue with pulling of docker images using Docker CLI

[mdziczkowski]

OS

• Distro: Debian
• Version codename: Trixie
• Docker version: 28.2.2

Description

After sucessfull logiin to docker from the CLI, I attemted to download one of the hardened images. Instead of download start, I had recieved an error displayed bellow:

docker: Error response from daemon: unknown: failed to resolve reference "dhi.io/openfga:latest": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://dhi.io/token?scope=repository%3Aopenfga%3Apull&service=registry.docker.io: 401 Unauthorized

Please verify the repositories with the hardened images afainst potential missconfigurations. Thanks

[Takadimi]

I don't think it's quite the same, but we're seeing intermittent failures with these errors:

ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Subsequent pulls can succeed in our CI with the same API key against the same image.

---

I can't imagine it's related, but I'll just note there is also this active incident being reported right now on https://dockerstatus.com

[cdupuis]

@Takadimi, we were to isolate that problem and implement a fix which was deployed a couple of hours ago. This problem should not occur again.

Thanks for reporting,

cd

[cdupuis]

@mdziczkowski, dhi.io does not support anonymous access which is what I believe this error message is indicating:

failed to fetch anonymous token:

Are you logging into dhi.io with docker login dhi.io by chance?

cd

[Takadimi]

Yes, it's logged in.

And actually, it looks like this may be an issue with a vendor (Depot). Builds/pulls are working when running direct docker commands, but not when those commands are managed by Depot.

So I think you can disregard this. Thanks for the response!

[Takadimi]

Hmm...I take that back. We just modified our CI pipelines to use docker directly (skipping Depot entirely) and it's failing intermittently with the error message I shared.

Logs

[+] Building 1.0s (3/3) FINISHED                                 docker:default
 => [internal] load build definition from Dockerfile                       0.1s
 => => transferring dockerfile: 1.01kB                                     0.0s
 => ERROR [internal] load metadata for dhi.io/python:3.11-alpine3.22-dev   0.8s
 => [auth] python:pull token for dhi.io                                    0.0s
------
 > [internal] load metadata for dhi.io/python:3.11-alpine3.22-dev:
------
Dockerfile:1
--------------------
   1 | >>> FROM dhi.io/python:3.11-alpine3.22-dev AS base
   2 |     
   3 |     WORKDIR /app
--------------------
ERROR: failed to build: failed to solve: dhi.io/python:3.11-alpine3.22-dev: failed to resolve source metadata for dhi.io/python:3.11-alpine3.22-dev: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Exited with code exit status 1

Is there any chance this is some kind of rate limiting issue that's surfacing this error, @cdupuis? We do have a number of images being built off the hardened base images in parallel. Not particularly high volume, but maybe it's a burst effect?

[cdupuis]

I'm not seeing any rate limit related errors in our logs. If you could narrow down the time window at bit more and perhaps from where you are pulling (Github) I can dig a bit more into our logs.

[Takadimi]

It's being run in CircleCI and one of the last failures was ~16:10:30 UTC

[kinoute]

I'm having similar issues through Harbor Proxy cache. I get randomly 404 not found errors on images.

[cdupuis]

@kinoute,

I think this error was fixed just a few hours ago. Please let me know if this happens again.

cd

[kinoute]

@cdupuis Works great now, thank you!

[mdziczkowski]

When I return back home (somewhere around Febuary), I'll check if the issue will still appear to me 🙂

[NKesar75]

@cdupuis, a few team members and I are noticing the issue more so recently when running it in AWS CodeBuild and locally running commands like

echo "<PAT_HERE>" | docker --debug login --username "<USERNAME_HERE>" --password-stdin dhi.io

Error response from daemon: Get "https://dhi.io/v2/": unauthorized: Authentication failed

As well as this issue below

Even with a successful message appearing, we are still encountering this issue.

[JoshBard]

+1. Still happening after clearing docker/config.json and logging in and out of dhi.io

[cdupuis]

Yes, we are aware and are looking into it.

cd

[JoshBard]

@cdupuis Thanks!

[NKesar75]

Should this issue be reopened until the issue is resolved again, so that if others run into the same issue, they are aware this is actively being looked into?

[cdupuis]

This was resolved 12 hours ago. We are not seeing any issues anymore.