Failing to use a dockerhub mirror with authentication

[sgandon]

Hello
We are hosting an artifactory server setup to mirror Dockerhub to avoid the rate limit issue and this mirror requires authentication.
We are using a runner version 2.301.1
We have setup the dockerRegistryMirror in our RunnerDeployment spec and checked that it is setup inside the pod in the /etc/docker/daemon.json.
We also are making sure that the /home/runner/.docker/config.json is setup with the mirror credentials.
Unfortunately when trying to pull an image from withing a Githubaction workflow like the following

jobs:
  test-mirror:
    runs-on: [self-hosted]
    steps:
      - name: test mirror
        run: docker pull alpine/helm:3.10.0-rc.1

it does not use the mirror, in fact looking in the runner logs we can see it tries to use the mirror but fails with this error

 time="2023-02-09T10:23:40.516028666Z" level=info msg="Attempting next endpoint for pull after error: Head \"https://docker-mirror.datatwn.com/v2/alpine/helm/manifests/3.10.0-rc.1\": unknown: Authentication is required" 

We have tried to shell into the container and we can do a direct download from the mirror successfully.

docker pull docker-mirror.datatwn.com/alpine/helm:3.10.0-rc.1
3.10.0-rc.1: Pulling from alpine/helm
9b18e9b68314: Downloading 

This is issue can be very annoying because it will eventually pull the image from dockerhub and we regularly face ratelimit issues and images cannot be dowloaded anymore. This is particularly true for public Github actions relying packaged using docker that we cannot update to use our direct registry.
Any help would be much appreciated.

[sgandon]

well after some research it seems that we are hitting this docker bug

[sgandon]

one possible workaround is here : moby/moby#30880 (comment)
with increased security here : moby/moby#30880 (comment)

[sgandon]

I confirm that those 2 workaround work and for those who are interested to make it work here is an excerpt of the RunnerDeployment

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: example-runnerdeploy
spec:
  template:
    spec:
      image: docker-io-remote.cd.ourcompany.com/summerwind/actions-runner-dind:v2.299.1-ubuntu-20.04-aa6dab5
      imagePullSecrets:
        - name: ourcompany-registry
      dockerEnabled: false
      dockerRegistryMirror: https://docker-io-remote.cd.ourcompany.com
      dockerdWithinRunnerContainer: true
      hostAliases: #this prevents our private registry creds (from the mirror-registry secret) to be leaked to docker servers
      - ip: "127.0.0.1"
        hostnames:
        - "index.docker.io"
        - "registry-1.docker.io"
        - "docker.io"


      initContainers: #this will copy the config.json file from the read-only mounted K8s secret to a read-write volume  
        - name: copy-docker-config
          image: docker-io-remote.cd.ourcompany.com/busybox:1.34.1
          command:
            - "sh"
            - "-c"
            - >
              set -x;
              cp /docker-conf-ro/config.json /docker-conf-rw/config.json;

          volumeMounts:
            - name: docker-conf-rw
              mountPath: /docker-conf-rw
            - name: github-actions-runner-registry
              mountPath: /docker-conf-ro/config.json
              subPath: .dockerconfigjson


      volumeMounts: #there is one of volume mounted in the runner with the rw docker config
        - name: docker-conf-rw
          mountPath: /home/runner/.docker/

      volumes:
        - name: docker-conf-rw
          emptyDir: {}
        - name: github-actions-runner-registry #this is mounting the secret containing the config.json file
          secret:
            secretName: mirror-registry

and of course the secret implementing the workaround for the docker issue called mirror-registry

  {"auths": {
      "https://index.docker.io/v1/": { "auth": "zekey" },
      "docker-io-remote.cd.ourcompany.com":{ "auth": "zekey" }
      }
  }

[Steed-LHV]

Thanks @sgandon, following your example but got this error
│ copy-docker-config cp: omitting directory '/docker-conf-ro/config.json'

[sgandon]

hello @Steed-LHV ,
This is most probably because your mirror-registry secret does not contain a file called .dockerconfigjson with the content of the docker config.

[Steed-LHV]

@sgandon exactly, the file calle config.json in my files. Already solved, much appreciated!