serviceAccountName causes RunnerDeployments to not work with pod phases?

[jrkarnes]

I wanted to ask a question here to rule out my doing anything wrong before posting an issue.

I am currently running an installation of ARC on 0.27.4 and I've come across a problem when I try and use the real-time runner statuses feature.

I have two runner deployments which are managed via ArgoCD:

Improved-Robots

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"actions.summerwind.dev/v1alpha1","kind":"RunnerDeployment","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"improved-robot-runners-non-prod"},"name":"improved-robot-runners-non-prod","namespace":"actions-runner-workers"},"spec":{"replicas":4,"template":{"spec":{"containers":[{"env":[{"name":"DISABLE_WAIT_FOR_DOCKER","value":"false"},{"name":"DOCKER_ENABLED","value":"true"},{"name":"DISABLE_WAIT_FOR_DOCKER","value":"false"},{"name":"DOCKER_ENABLED","value":"true"}],"livenessProbe":{"exec":{"command":["/bin/bash","-c","ls
      -l
      /runner/_diag/Worker*"]},"failureThreshold":5,"initialDelaySeconds":1800,"periodSeconds":120,"timeoutSeconds":30},"name":"runner","resources":{"limits":{"memory":"4Gi"},"requests":{"cpu":"0.5","memory":"4Gi"}}}],"dockerRegistryMirror":"http://twuni-docker-registry.docker-registry-mirror.svc.cluster.local:5000","env":[{"name":"DISABLE_WAIT_FOR_DOCKER","value":"false"},{"name":"DOCKER_ENABLED","value":"true"}],"ephemeral":true,"image":"ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:v2.307.1-ubuntu-20.04-dcb64f0@sha256:2917c1271f7cdee3685846a2dc7f1f4c66e8625f0ec76083639d2d70b8743752","imagePullPolicy":"IfNotPresent","labels":["improved-robot-builder-np","us-east-1"],"repository":"ETS-CloudAutomation/improved-robot","securityContext":{"fsGroup":1000},"serviceAccountName":"gherunners-non-prod-cd-ghe-irsa","volumeMounts":[{"mountPath":"/dev/shm","name":"dshm"}],"volumes":[{"emptyDir":{"medium":"Memory"},"name":"dshm"}]}}}}
  creationTimestamp: '2023-08-02T20:43:01Z'
  generation: 5
  labels:
    argocd.argoproj.io/instance: improved-robot-runners-non-prod
  managedFields:
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:argocd.argoproj.io/instance: {}
        f:spec:
          .: {}
          f:replicas: {}
          f:template:
            .: {}
            f:spec:
              .: {}
              f:containers: {}
              f:dockerRegistryMirror: {}
              f:env: {}
              f:ephemeral: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:labels: {}
              f:repository: {}
              f:securityContext:
                .: {}
                f:fsGroup: {}
              f:serviceAccountName: {}
              f:volumeMounts: {}
              f:volumes: {}
      manager: argocd-application-controller
      operation: Update
      time: '2023-08-08T14:47:09Z'
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:availableReplicas: {}
          f:desiredReplicas: {}
          f:readyReplicas: {}
          f:replicas: {}
          f:updatedReplicas: {}
      manager: manager
      operation: Update
      subresource: status
      time: '2023-09-01T15:52:16Z'
  name: improved-robot-runners-non-prod
  namespace: actions-runner-workers
  resourceVersion: '43824232'
  uid: f868d12a-fba7-4a9e-9dd3-2681dba43ecf
  selfLink: >-
    /apis/actions.summerwind.dev/v1alpha1/namespaces/actions-runner-workers/runnerdeployments/improved-robot-runners-non-prod
status:
  availableReplicas: 4
  desiredReplicas: 4
  readyReplicas: 4
  replicas: 4
  updatedReplicas: 4
spec:
  effectiveTime: null
  replicas: 4
  selector: null
  template:
    metadata: {}
    spec:
      containers:
        - env:
            - name: DISABLE_WAIT_FOR_DOCKER
              value: 'false'
            - name: DOCKER_ENABLED
              value: 'true'
            - name: DISABLE_WAIT_FOR_DOCKER
              value: 'false'
            - name: DOCKER_ENABLED
              value: 'true'
          livenessProbe:
            exec:
              command:
                - /bin/bash
                - '-c'
                - ls -l /runner/_diag/Worker*
            failureThreshold: 5
            initialDelaySeconds: 1800
            periodSeconds: 120
            timeoutSeconds: 30
          name: runner
          resources:
            limits:
              memory: 4Gi
            requests:
              cpu: 500m
              memory: 4Gi
      dockerRegistryMirror: >-
        http://twuni-docker-registry.docker-registry-mirror.svc.cluster.local:5000
      dockerdContainerResources: {}
      env:
        - name: DISABLE_WAIT_FOR_DOCKER
          value: 'false'
        - name: DOCKER_ENABLED
          value: 'true'
      ephemeral: true
      image: >-
        ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:v2.307.1-ubuntu-20.04-dcb64f0@sha256:2917c1271f7cdee3685846a2dc7f1f4c66e8625f0ec76083639d2d70b8743752
      imagePullPolicy: IfNotPresent
      labels:
        - improved-robot-builder-np
        - us-east-1
      repository: ETS-CloudAutomation/improved-robot
      resources: {}
      securityContext:
        fsGroup: 1000
      serviceAccountName: gherunners-non-prod-cd-ghe-irsa
      volumeMounts:
        - mountPath: /dev/shm
          name: dshm
      volumes:
        - emptyDir:
            medium: Memory
          name: dshm

and Enterprise Testing:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"actions.summerwind.dev/v1alpha1","kind":"RunnerDeployment","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"ent-non-prod-us-east-1"},"name":"ent-non-prod-us-east-1","namespace":"actions-runner-workers"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"DISABLE_WAIT_FOR_DOCKER","value":"false"},{"name":"DOCKER_ENABLED","value":"true"},{"name":"DISABLE_WAIT_FOR_DOCKER","value":"false"},{"name":"DOCKER_ENABLED","value":"true"}],"livenessProbe":{"exec":{"command":["/bin/bash","-c","ls
      -l
      /runner/_diag/Worker*"]},"failureThreshold":5,"initialDelaySeconds":1800,"periodSeconds":120,"timeoutSeconds":30},"name":"runner","resources":{"limits":{"memory":"8Gi"},"requests":{"cpu":"0.5","memory":"8Gi"}}}],"dockerRegistryMirror":"http://twuni-docker-registry.docker-registry-mirror.svc.cluster.local:5000","enterprise":"<REDACTED>","env":[{"name":"DISABLE_WAIT_FOR_DOCKER","value":"false"},{"name":"DOCKER_ENABLED","value":"true"}],"ephemeral":true,"group":"Test
      Enterprise
      Runners","image":"ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:v2.307.1-ubuntu-20.04-dcb64f0@sha256:2917c1271f7cdee3685846a2dc7f1f4c66e8625f0ec76083639d2d70b8743752","imagePullPolicy":"IfNotPresent","labels":["Enterprise-Testing-np","us-east-1"],"volumeMounts":[{"mountPath":"/dev/shm","name":"dshm"}],"volumes":[{"emptyDir":{"medium":"Memory"},"name":"dshm"}]}}}}
  creationTimestamp: '2023-08-02T20:43:01Z'
  generation: 102
  labels:
    argocd.argoproj.io/instance: ent-non-prod-us-east-1
  managedFields:
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:argocd.argoproj.io/instance: {}
        f:spec:
          .: {}
          f:template:
            .: {}
            f:spec:
              .: {}
              f:containers: {}
              f:dockerRegistryMirror: {}
              f:enterprise: {}
              f:env: {}
              f:ephemeral: {}
              f:group: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:labels: {}
              f:volumeMounts: {}
              f:volumes: {}
      manager: argocd-application-controller
      operation: Update
      time: '2023-08-08T14:41:17Z'
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:replicas: {}
      manager: manager
      operation: Update
      time: '2023-09-01T18:41:09Z'
    - apiVersion: actions.summerwind.dev/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:availableReplicas: {}
          f:desiredReplicas: {}
          f:readyReplicas: {}
          f:replicas: {}
          f:updatedReplicas: {}
      manager: manager
      operation: Update
      subresource: status
      time: '2023-09-01T19:11:45Z'
  name: ent-non-prod-us-east-1
  namespace: actions-runner-workers
  resourceVersion: '43880348'
  uid: 99a87885-b2a2-49e7-a57b-e74e0df24eac
  selfLink: >-
    /apis/actions.summerwind.dev/v1alpha1/namespaces/actions-runner-workers/runnerdeployments/ent-non-prod-us-east-1
status:
  availableReplicas: 2
  desiredReplicas: 2
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2
spec:
  effectiveTime: null
  replicas: 2
  selector: null
  template:
    metadata: {}
    spec:
      containers:
        - env:
            - name: DISABLE_WAIT_FOR_DOCKER
              value: 'false'
            - name: DOCKER_ENABLED
              value: 'true'
            - name: DISABLE_WAIT_FOR_DOCKER
              value: 'false'
            - name: DOCKER_ENABLED
              value: 'true'
          livenessProbe:
            exec:
              command:
                - /bin/bash
                - '-c'
                - ls -l /runner/_diag/Worker*
            failureThreshold: 5
            initialDelaySeconds: 1800
            periodSeconds: 120
            timeoutSeconds: 30
          name: runner
          resources:
            limits:
              memory: 8Gi
            requests:
              cpu: 500m
              memory: 8Gi
      dockerRegistryMirror: >-
        http://twuni-docker-registry.docker-registry-mirror.svc.cluster.local:5000
      dockerdContainerResources: {}
      enterprise: <REDACTED>
      env:
        - name: DISABLE_WAIT_FOR_DOCKER
          value: 'false'
        - name: DOCKER_ENABLED
          value: 'true'
      ephemeral: true
      group: Test Enterprise Runners
      image: >-
        ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:v2.307.1-ubuntu-20.04-dcb64f0@sha256:2917c1271f7cdee3685846a2dc7f1f4c66e8625f0ec76083639d2d70b8743752
      imagePullPolicy: IfNotPresent
      labels:
        - Enterprise-Testing-np
        - us-east-1
      resources: {}
      volumeMounts:
        - mountPath: /dev/shm
          name: dshm
      volumes:
        - emptyDir:
            medium: Memory
          name: dshm

The Enterprise Test runners work with the pod phases as expected because the volumeMounts contain a volume mounting @ /var/run/secrets/kubernetes.io/serviceaccount:

apiVersion: v1
kind: Pod
metadata:
  name: ent-non-prod-us-east-1-67lxz-4592n
  namespace: actions-runner-workers
  uid: c36ea560-3dd0-4278-b9ff-61e7f928af2d
  resourceVersion: '43829509'
  creationTimestamp: '2023-09-01T16:10:58Z'
  labels:
    actions-runner: ''
    actions-runner-controller/inject-registration-token: 'true'
    pod-template-hash: 7dc7984f4
    runner-deployment-name: ent-non-prod-us-east-1
    runner-template-hash: 7956c56c49
  annotations:
    actions-runner-controller/token-expires-at: '2023-09-01T13:08:30-04:00'
    actions-runner/id: '5636785'
    sync-time: '2023-09-01T16:10:58Z'
  ownerReferences:
    - apiVersion: actions.summerwind.dev/v1alpha1
      kind: Runner
      name: ent-non-prod-us-east-1-67lxz-4592n
      uid: 325eaa40-f5fe-4be6-ad06-4538ed34769d
      controller: true
      blockOwnerDeletion: true
  finalizers:
    - actions.summerwind.dev/runner-pod
  selfLink: >-
    /api/v1/namespaces/actions-runner-workers/pods/ent-non-prod-us-east-1-67lxz-4592n
status:
  phase: Running
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-09-01T16:10:58Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-09-01T16:10:59Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-09-01T16:10:59Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-09-01T16:10:58Z'
  hostIP: 100.82.15.166
  podIP: 100.82.15.135
  podIPs:
    - ip: 100.82.15.135
  startTime: '2023-09-01T16:10:58Z'
  containerStatuses:
    - name: docker
      state:
        running:
          startedAt: '2023-09-01T16:10:59Z'
      lastState: {}
      ready: true
      restartCount: 0
      image: docker.io/library/docker:dind
      imageID: >-
        docker.io/library/docker@sha256:2985399fb735fa4c3b6cfba8d1087f011dc4c6f5fde105c9c5a77be472b73836
      containerID: >-
        containerd://37abede89ed89562c52771ba5d34770caac927792764248c2e808ff04408c4ff
      started: true
    - name: runner
      state:
        running:
          startedAt: '2023-09-01T16:10:59Z'
      lastState: {}
      ready: true
      restartCount: 0
      image: sha256:c6d3422333464fcb3ad46c52b602d7715135bbb85df6cbd038f0fc1cf0ba5be0
      imageID: >-
        ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner@sha256:2917c1271f7cdee3685846a2dc7f1f4c66e8625f0ec76083639d2d70b8743752
      containerID: >-
        containerd://6001aaf73e37e20fd2efe32d16fbf9f4cb4e3f7c2f2c94f616a0ac4634594e81
      started: true
  qosClass: Burstable
spec:
  volumes:
    - name: runner
      emptyDir: {}
    - name: work
      emptyDir: {}
    - name: var-run
      emptyDir:
        medium: Memory
        sizeLimit: 1M
    - name: dshm
      emptyDir:
        medium: Memory
    - name: kube-api-access-l5jf6
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: runner
      image: >-
        ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:v2.307.1-ubuntu-20.04-dcb64f0@sha256:2917c1271f7cdee3685846a2dc7f1f4c66e8625f0ec76083639d2d70b8743752
      env:
        - name: DISABLE_WAIT_FOR_DOCKER
          value: 'false'
        - name: DOCKER_ENABLED
          value: 'true'
        - name: DISABLE_WAIT_FOR_DOCKER
          value: 'false'
        - name: DOCKER_ENABLED
          value: 'true'
        - name: RUNNER_ORG
        - name: RUNNER_REPO
        - name: RUNNER_ENTERPRISE
          value: <REDACTED>
        - name: RUNNER_LABELS
          value: Enterprise-Testing-np,us-east-1
        - name: RUNNER_GROUP
          value: Test Enterprise Runners
        - name: DOCKER_ENABLED
          value: 'true'
        - name: DOCKERD_IN_RUNNER
          value: 'false'
        - name: GITHUB_URL
          value: https://<REDACTED>/
        - name: RUNNER_WORKDIR
          value: /runner/_work
        - name: RUNNER_EPHEMERAL
          value: 'true'
        - name: RUNNER_STATUS_UPDATE_HOOK
          value: 'false'
        - name: GITHUB_ACTIONS_RUNNER_EXTRA_USER_AGENT
          value: actions-runner-controller/v0.27.4
        - name: DOCKER_HOST
          value: unix:///run/docker.sock
        - name: RUNNER_NAME
          value: ent-non-prod-us-east-1-67lxz-4592n
        - name: RUNNER_TOKEN
          value: <REDACTED>
      resources:
        limits:
          memory: 8Gi
        requests:
          cpu: 500m
          memory: 8Gi
      volumeMounts:
        - name: runner
          mountPath: /runner
        - name: work
          mountPath: /runner/_work
        - name: var-run
          mountPath: /run
        - name: dshm
          mountPath: /dev/shm
        - name: kube-api-access-l5jf6
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      livenessProbe:
        exec:
          command:
            - /bin/bash
            - '-c'
            - ls -l /runner/_diag/Worker*
        initialDelaySeconds: 1800
        timeoutSeconds: 30
        periodSeconds: 120
        successThreshold: 1
        failureThreshold: 5
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext: {}
    - name: docker
      image: docker:dind
      args:
        - dockerd
        - '--host=unix:///run/docker.sock'
        - '--group=$(DOCKER_GROUP_GID)'
        - >-
          --registry-mirror=http://twuni-docker-registry.docker-registry-mirror.svc.cluster.local:5000
      env:
        - name: DOCKER_GROUP_GID
          value: '1001'
      resources: {}
      volumeMounts:
        - name: runner
          mountPath: /runner
        - name: var-run
          mountPath: /run
        - name: work
          mountPath: /runner/_work
        - name: kube-api-access-l5jf6
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - '-c'
              - >-
                timeout "${RUNNER_GRACEFUL_STOP_TIMEOUT:-15}" /bin/sh -c "echo
                'Prestop hook started'; while [ -f /runner/.runner ]; do sleep
                1; done; echo 'Waiting for dockerd to start'; while ! pgrep -x
                dockerd; do sleep 1; done; echo 'Prestop hook stopped'"
                >/proc/1/fd/1 2>&1
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        privileged: true
  restartPolicy: Never
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: default
  serviceAccount: default
  nodeName: ip-100-82-15-166.ec2.internal
  securityContext: {}
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

but the Improved-Robots runners DO NOT:

apiVersion: v1
kind: Pod
metadata:
  name: improved-robot-runners-non-prod-6756g-br7s2
  namespace: actions-runner-workers
  uid: 77fd27bf-e0dc-45f0-bcc8-aa0bd47b94a3
  resourceVersion: '43823904'
  creationTimestamp: '2023-09-01T15:50:44Z'
  labels:
    actions-runner: ''
    actions-runner-controller/inject-registration-token: 'true'
    pod-template-hash: 5cb6fbfbdc
    runner-deployment-name: improved-robot-runners-non-prod
    runner-template-hash: 69f6c88b4d
  annotations:
    actions-runner-controller/token-expires-at: '2023-09-01T12:50:23-04:00'
    actions-runner/id: '98942'
    sync-time: '2023-09-01T15:50:44Z'
  ownerReferences:
    - apiVersion: actions.summerwind.dev/v1alpha1
      kind: Runner
      name: improved-robot-runners-non-prod-6756g-br7s2
      uid: ce3d9e6a-c2b7-467f-a223-2f9bd48ea3b4
      controller: true
      blockOwnerDeletion: true
  finalizers:
    - actions.summerwind.dev/runner-pod
  selfLink: >-
    /api/v1/namespaces/actions-runner-workers/pods/improved-robot-runners-non-prod-6756g-br7s2
status:
  phase: Running
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-09-01T15:50:44Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-09-01T15:50:45Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-09-01T15:50:45Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-09-01T15:50:44Z'
  hostIP: 100.82.15.22
  podIP: 100.82.15.17
  podIPs:
    - ip: 100.82.15.17
  startTime: '2023-09-01T15:50:44Z'
  containerStatuses:
    - name: docker
      state:
        running:
          startedAt: '2023-09-01T15:50:45Z'
      lastState: {}
      ready: true
      restartCount: 0
      image: docker.io/library/docker:dind
      imageID: >-
        docker.io/library/docker@sha256:25a7feece7050334e8bd478dc9b6031c24db7fe81b2665abe690698ec52074f2
      containerID: >-
        containerd://2f4d8407f8d177a27353e20228060b69cae1159ac1473297ea221ff408ee0d1c
      started: true
    - name: runner
      state:
        running:
          startedAt: '2023-09-01T15:50:45Z'
      lastState: {}
      ready: true
      restartCount: 0
      image: sha256:c6d3422333464fcb3ad46c52b602d7715135bbb85df6cbd038f0fc1cf0ba5be0
      imageID: >-
        ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner@sha256:2917c1271f7cdee3685846a2dc7f1f4c66e8625f0ec76083639d2d70b8743752
      containerID: >-
        containerd://33981dc0b901d9348c61d95fac5b16d2dd57760d982ef62b714e058ce7f33444
      started: true
  qosClass: Burstable
spec:
  volumes:
    - name: aws-iam-token
      projected:
        sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token
        defaultMode: 420
    - name: runner
      emptyDir: {}
    - name: work
      emptyDir: {}
    - name: var-run
      emptyDir:
        medium: Memory
        sizeLimit: 1M
    - name: dshm
      emptyDir:
        medium: Memory
  containers:
    - name: runner
      image: >-
        ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:v2.307.1-ubuntu-20.04-dcb64f0@sha256:2917c1271f7cdee3685846a2dc7f1f4c66e8625f0ec76083639d2d70b8743752
      env:
        - name: DISABLE_WAIT_FOR_DOCKER
          value: 'false'
        - name: DOCKER_ENABLED
          value: 'true'
        - name: DISABLE_WAIT_FOR_DOCKER
          value: 'false'
        - name: DOCKER_ENABLED
          value: 'true'
        - name: RUNNER_ORG
        - name: RUNNER_REPO
          value: ETS-CloudAutomation/improved-robot
        - name: RUNNER_ENTERPRISE
        - name: RUNNER_LABELS
          value: improved-robot-builder-np,us-east-1
        - name: RUNNER_GROUP
        - name: DOCKER_ENABLED
          value: 'true'
        - name: DOCKERD_IN_RUNNER
          value: 'false'
        - name: GITHUB_URL
          value: https://<REDACTED>
        - name: RUNNER_WORKDIR
          value: /runner/_work
        - name: RUNNER_EPHEMERAL
          value: 'true'
        - name: RUNNER_STATUS_UPDATE_HOOK
          value: 'false'
        - name: GITHUB_ACTIONS_RUNNER_EXTRA_USER_AGENT
          value: actions-runner-controller/v0.27.4
        - name: DOCKER_HOST
          value: unix:///run/docker.sock
        - name: RUNNER_NAME
          value: improved-robot-runners-non-prod-6756g-br7s2
        - name: RUNNER_TOKEN
          value: <REDACTED>
        - name: AWS_STS_REGIONAL_ENDPOINTS
          value: regional
        - name: AWS_DEFAULT_REGION
          value: us-east-1
        - name: AWS_REGION
          value: us-east-1
        - name: AWS_ROLE_ARN
          value: >-
            arn:aws:iam::<AWS_ACCOUNT_ID>:role/acct-managed/gherunners-non-prod-cd-ghe-irsa
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      resources:
        limits:
          memory: 4Gi
        requests:
          cpu: 500m
          memory: 4Gi
      volumeMounts:
        - name: runner
          mountPath: /runner
        - name: work
          mountPath: /runner/_work
        - name: var-run
          mountPath: /run
        - name: dshm
          mountPath: /dev/shm
        - name: aws-iam-token
          readOnly: true
          mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      livenessProbe:
        exec:
          command:
            - /bin/bash
            - '-c'
            - ls -l /runner/_diag/Worker*
        initialDelaySeconds: 1800
        timeoutSeconds: 30
        periodSeconds: 120
        successThreshold: 1
        failureThreshold: 5
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext: {}
    - name: docker
      image: docker:dind
      args:
        - dockerd
        - '--host=unix:///run/docker.sock'
        - '--group=$(DOCKER_GROUP_GID)'
        - >-
          --registry-mirror=http://twuni-docker-registry.docker-registry-mirror.svc.cluster.local:5000
      env:
        - name: DOCKER_GROUP_GID
          value: '1001'
        - name: AWS_STS_REGIONAL_ENDPOINTS
          value: regional
        - name: AWS_DEFAULT_REGION
          value: us-east-1
        - name: AWS_REGION
          value: us-east-1
        - name: AWS_ROLE_ARN
          value: >-
            arn:aws:iam::<AWS_ACCOUNT_ID>:role/acct-managed/gherunners-non-prod-cd-ghe-irsa
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
      resources: {}
      volumeMounts:
        - name: runner
          mountPath: /runner
        - name: var-run
          mountPath: /run
        - name: work
          mountPath: /runner/_work
        - name: aws-iam-token
          readOnly: true
          mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - '-c'
              - >-
                timeout "${RUNNER_GRACEFUL_STOP_TIMEOUT:-15}" /bin/sh -c "echo
                'Prestop hook started'; while [ -f /runner/.runner ]; do sleep
                1; done; echo 'Waiting for dockerd to start'; while ! pgrep -x
                dockerd; do sleep 1; done; echo 'Prestop hook stopped'"
                >/proc/1/fd/1 2>&1
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        privileged: true
  restartPolicy: Never
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: gherunners-non-prod-cd-ghe-irsa
  serviceAccount: gherunners-non-prod-cd-ghe-irsa
  nodeName: ip-100-82-15-22.ec2.internal
  securityContext:
    fsGroup: 1000
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

The lack of a kube-api-access volumeMount with accompanying data is causing the pod phase startup hooks to fail because of an expectation that the kubernetes.io secret will exist at /var/run here.

I've looked through the runner code as best as I can and I don't see anything that explicitly mounts this information. The only differences between the deployments are:

• One targets a specific repo and the other is scoped to the enterprise
• The repo specific fleet uses a serviceAccount that is not default and the enterprise fleet uses the default one.

I was really hoping that someone would be able to point out or explain why this behavior is happening and if it should be happening. I consider myself fairly well versed in k8s but this one has me stumped.

Any help is welcome!

[jrkarnes]

I'm closing this because it was discovered during some followup investigation that the automountServiceAccountToken: false property was set on the serviceAccount for the improved-robot fleet.

This is the root cause of the error and changing that property to true has fixed our problems.