Job container authentication to a private Google Container Registry

[jorob16]

I'm trying to give my runner access to a private Docker registry implemented using Google Container Registry. I need to be able to pull images that are on this Registry server in order to use them for job containers used in my workflows. I'm running the actions runner controller on Google Kubernetes Engine and it's all working fine. IAM is assigned through Workload Identity.

The problem: job containers that need images from private Docker registries only accept usernames and passwords (See: https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container). For Google Container Registry it's best practice to install the gcloud sdk and run the gcloud credential helper using gcloud auth configure-docker in order to retrieve short lived access tokens for Docker authentication (See: https://cloud.google.com/container-registry/docs/advanced-authentication).

I've set up a RunnerDeployment called oos-basic which uses a modified actions-runner image that has the gcloud sdk included. And I'm trying to run the following workflow:

name: Create new release version
on:
  workflow_dispatch:
    inputs:
      release-version:
        description: Release version, leave empty for default
        default: ''
        type: string
jobs:
  update-version:
    name: Update release version
    runs-on: [self-hosted, oos-basic]
    container: eu.gcr.io/*<redacted>*/cicd-bumpversion:1
    steps:
    ...

Has anyone worked on a problem similar to this? Ideally I would run gcloud auth configure-docker on the runner before the actual job starts. Right now I'm obviously getting an authentication error:

I've already tried adding a postStart lifecycle hook to the runner pod spec that runs a bash script that simply runs the command, but that kills my runner right after startup with the following error: unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory

So I figured that I would have to wait for Docker to have retrieved its' certs, so I added a simple loop in the script that waits for ca.pem to be available:

timeout=60
passed=0
echo "Waiting for docker daemon to be fully started"
until [ -f "${DOCKER_CERT_PATH}/ca.pem" ] || [ ${timeout} -lt ${passed} ]; do
    echo -n "."
    passed=`expr $passed + 1`
    sleep 1
done
echo "Adding docker credentials for Google Container Registry"
gcloud auth configure-docker

Sadly that still gives the same error.

Would love to hear if anyone has some suggestions for how to best tackle this problem or whether it's at all possible to solve!

[roulettedares]

i created an internal composite action for this

name: Build and Push Container Image
description: 'Build and Push Container Image'
inputs:
  artifact:
    description: name of a github artifact to download
    required: false
  tags:
    description: docker image tags to push
    required: true
outputs:
  image:
    description: "The docker image"
    value: ${{ steps.container-build.outputs.imageid }}
runs:
  using: "composite"
  steps:
  - name: Set up Docker Context for Buildx
    shell: bash
    run: docker context create arc
  - uses: docker/setup-buildx-action@f03ac48505955848960e80bbb68046aa35c7b9e7 # v2.4.1
    with:
      endpoint: arc
  - name: Download artifact
    if: ${{ inputs.artifact != '' }}
    uses: actions/download-artifact@v3
    with:
      name: ${{ inputs.artifact }}
  - name: gcr-auth
    shell: bash
    id: gcr-auth
    run: |-
      access_token="$(curl -s --retry 5 \
        "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
        -H "Metadata-Flavor: Google" \
        | jq -r .access_token)"
      echo "access_token=$access_token" >> $GITHUB_OUTPUT
      echo "::add-mask::$access_token"
  - name: Login to GCR
    uses: docker/login-action@v2
    with:
      registry: gcr.io
      username: oauth2accesstoken
      password: ${{ steps.gcr-auth.outputs.access_token }}
  - name: Build and push Docker images
    id: container-build
    uses: docker/build-push-action@3b5e8027fcad23fda98b2e3ac259d8d67585f671 #v4.0.0
    with:
      push: true
      context: .
      tags: |
        ${{ inputs.tags }}%

[jorob16]

That won't work since job container images are pulled before any actions are actually run.

I already fixed it by creating a custom actions runner image that has the docker-credential-gcr binary preinstalled. That way docker automatically requests access tokens from GCR whenever a docker command is invoked.

[jorob16]

For future reference, I got around to fixing this for myself, and my chosen solution was creating a custom actions runner image that has docker-credential-gcr preinstalled.

Dockerfile

FROM summerwind/actions-runner:v2.299.1-ubuntu-20.04
ARG CRED_HELPER_VERSION=2.1.6
ARG CRED_HELPER_TARBALL=docker-credential-gcr_linux_amd64-${CRED_HELPER_VERSION}.tar.gz
USER root
RUN wget https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v${CRED_HELPER_VERSION}/${CRED_HELPER_TARBALL} && \
    mkdir helper && tar -xzf ${CRED_HELPER_TARBALL} -C ./helper && \
    mv ./helper/docker-credential-gcr /usr/bin/ && rm -r helper && rm ${CRED_HELPER_TARBALL}
COPY --chown=runner dockerconfig.json /home/runner/.docker/config.json
USER runner

dockerconfig.json

{
    "credHelpers": {
        "gcr.io": "gcr",
        "asia.gcr.io": "gcr",
        "eu.gcr.io": "gcr",
        "us.gcr.io": "gcr"
    }
}

By using this image all docker command invocations automatically pull access tokens from GCR when any of the GCR hostnames are referenced in the image registry name. Now assigning the proper Container Registry IAM roles to your service accounts is enough.

[AlfonsoUceda]

Worked for me too, thanks!

[SamyDjemai]

Thanks, your solution works perfectly!

[gr8jen]

It looks promising, I was having the same problem with using Github hosted runners. But these custom runners are now running at your own cluster?
How did you solve the problems with sharing the volume between the runner and the worker? For as far as I know there is a challenge when you have to schedule those 2 pods at the same node when using kubernetes mode instead of dind. The few first will work, but sometimes there is not enough space en the worker pod will be placed at the next node. And that the volume can not be shared between nodes. Could you tell me more about how you run these runners?