Authenticated Docker Hub Mirror Problem

[brunoabreu0]

Image I'm using:
bottlerocket-aws-k8s-1.32-x86_64-v1.30*

What I expected to happen:
Pull images from my authenticated docker-hub mirror.

What actually happened:
403 error

How to reproduce the problem:
I have setup my docker hub mirror and its credentials in the toml userdata, and when I create pods that references the mirror directly, the images are pulled properly (the authentication works fine). However, when I do not reference the images directly (so they can use the mirror), the node tries to pull from mirror correctly, however, without the credentials:

This is my toml userdata:

[[settings.container-registry.mirrors]]
registry = "docker.io"
endpoint = ["https://my-mirrors.com/docker-hub"]

[[settings.container-registry.credentials]]
registry = "my-mirrors.com"
username = "username"
password = "*****"
The error I get when trying to pull the image image: natsio/nats-server-config-reloader:0.14.0 is:

Failed to pull image "natsio/nats-server-config-reloader:0.14.0": failed to pull and unpack image "docker.io/natsio/nats-server-config-reloader:0.14.0": failed to resolve reference "docker.io/natsio/nats-server-config-reloader:0.14.0": unexpected status from HEAD request to https://my-mirrors.com/docker-hub/natsio/nats-server-config-reloader/manifests/0.14.0?ns=docker.io: 403
Since the error mentions my setup mirror, I assume the mirror configuration at [[settings.container-registry.mirrors]] is working fine, despite not getting the authentication configuration from [[settings.container-registry.credentials]]

If I setup the pod to pull the image image: my-mirros.com/docker-hub/natsio/nats-server-config-reloader:0.14.0, the image is pulled successfully, which makes me assume that the configuration set at [[settings.container-registry.credentials]] is also correct, despite the mirror configuration not working together.

[vigh-m]

Hi,
I tested this with a private github registry. My working config was as follows:

# apiclient get settings.container-registry
{
  "settings": {
    "container-registry": {
      "credentials": [
        {
          "password": "<personal-access-token>",
          "registry": "ghcr.io",
          "username": "<username>"
        }
      ],
      "mirrors": [
        {
          "endpoint": [
            "ghcr.io"
          ],
          "registry": "ghcr.io"
        }
      ]
    }
  }
}

which transalates to the following in the containerd/config.toml

# sheltie cat /etc/containerd/config.toml
.
.
.
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."ghcr.io"]
endpoint = ["ghcr.io"]

[plugins."io.containerd.grpc.v1.cri".registry.configs."ghcr.io".auth]
username = "<username>"
password = "<personal-access-token>"

From here, running kubectl -n default run -it --image ghcr.io/<username>/<private-repo>/alpine:latest test was able to pull the latest image in my private registry.

I think you're running into this issue where kubelet is unable to translate the image registry to the mirrored url. So, using the full URL is the recommended approach (my-mirrors.com vs docker.io in your case)

Along with this, the registry for endpoint and credentials should be the same.