ContainerMode kubernetes and private docker registrys fail to pull image

[robmaxwellirl]

When I'm using ARC with a Runner Deployment for containers with different labels, when trying to pull containers from a private registry with it, it's failing with ImagePullBackOff.

This is running off a VMWare Tanzu cluster, and I've obfuscated away any private information below.

When I switch up the actions YAML to point to a github hosted (ubuntu-latest) runner it works fine.

The RunnerDeployment looks like

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: my-runner-container-deployment
  namespace: default
spec:
  replicas: 5
  template:
    spec:
      organization: myorg
      group: myorg_self_hosted
      ephemeral: false
      dockerMTU: 1450
      containerMode: kubernetes
      serviceAccountName: runner-service-account
      labels:
        - myorg-runner-container
      workVolumeClaimTemplate:
        storageClassName: "tanzu-storage-policy"
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

The actions YAML

name: 'Run CI'

on:
  workflow_call:

jobs:
  test_haproxy_setup:
    runs-on: myorg-runner-container
    container:
      image: quay.io/private/ansible_test:ubuntu
      credentials:
        username: myquay+creds
        password: supersecretpass
    steps:
    - uses: actions/checkout@v3

    - uses: actions/setup-python@v4
      with:
        python-version: '3.10'

    - name: Install python dependencies
      run: pip install -r requirements/2.10.6/default.txt -r requirements/2.10.6/testing.txt

The resultant output of the workflow pod.

Name:         myorg-runner-container-deployment-qfdf7-mj2bm-workflow
Namespace:    default
Priority:     0
Node:         github-actions-workers-rdmjv-58b74c545-7mpln/10.40.102.49
Start Time:   Tue, 19 Jul 2022 11:39:27 +0100
Labels:       runner-pod=myorg-runner-container-deployment-qfdf7-mj2bm
Annotations:  kubernetes.io/psp: vmware-system-privileged
Status:       Pending
IP:           192.168.2.53
IPs:
  IP:  192.168.2.53
Containers:
  job:
    Container ID:  
    Image:         quay.io/private/ansible_test:ubuntu
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      tail
    Args:
      -f
      /dev/null
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /__e from work (rw,path="externals")
      /__w from work (rw)
      /github/home from work (rw,path="_temp/_github_home")
      /github/workflow from work (rw,path="_temp/_github_workflow")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6kdvv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  work:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  myorg-runner-container-deployment-qfdf7-mj2bm-work
    ReadOnly:   false
  kube-api-access-6kdvv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                From     Message
  ----     ------   ----               ----     -------
  Normal   Pulling  14s (x3 over 53s)  kubelet  Pulling image "quay.io/private/ansible_test:ubuntu"
  Warning  Failed   14s (x3 over 53s)  kubelet  Failed to pull image "quay.io/private/ansible_test:ubuntu": rpc error: code = Unknown desc = failed to pull and unpack image "quay.io/private/ansible_test:ubuntu": failed to resolve reference "quay.io/udemy/ansible_test:ubuntu": unexpected status code [manifests ubuntu]: 401 UNAUTHORIZED
  Warning  Failed   14s (x3 over 53s)  kubelet  Error: ErrImagePull
  Normal   BackOff  3s (x3 over 52s)   kubelet  Back-off pulling image "quay.io/private/ansible_test:ubuntu"
  Warning  Failed   3s (x3 over 52s)   kubelet  Error: ImagePullBackOff

Any help will be greatly appreciated here.

[acutillas-gs]

Did you ever solve this? I have the same issue.

[robmaxwellirl]

hey,

I ended up opening a support issue on the actual actions runner-container-hook to add support for it.

actions/runner-container-hooks#24

It would seem using an updated version of the runner controller would use this, but never actually got around to testing it out as of yet.