Runners don't clean up after themselves on NVMe storage despite ephemeral: true

[sarahkadar]

We have the following setup:

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: contoso-github-runners-arm
  namespace: github-runners
spec:
  template:
    spec:
      env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
      organization: contoso
      dockerRegistryMirror: https://mirror.gcr.io/
      nodeSelector:
        kubernetes.io/arch: arm64
      tolerations:
      - key: "nvme-ssd-enabled"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
      dockerVolumeMounts:
      - mountPath: /var/lib/docker
        name: docker
        subPathExpr: $(POD_NAME)-docker
      - mountPath: /runner/_work
        name: work
        subPathExpr: $(POD_NAME)-work
      volumeMounts:
      - mountPath: /runner/_work
        name: work
        subPathExpr: $(POD_NAME)-work
      # - mountPath: /tmp
      #   name: tmp
      #   subPathExpr: $(POD_NAME)-tmp
      dockerEnv:
      - name: POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      volumes:
      - hostPath:
          path: /nvme/disk
        name: docker
      - hostPath:
          path: /nvme/disk
        name: work
      # - hostPath:
      #     path: /nvme/disk
      #   name: tmp
      ephemeral: true
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: contoso-github-runners-autoscaler-arm
  namespace: github-runners
spec:
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    name: contoso-github-runners-arm
  minReplicas: 6
  maxReplicas: 24
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.50'
    scaleDownThreshold: '0.25'
    scaleUpFactor: '3'
    scaleDownFactor: '0.5'
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: contoso-github-runners-x64-fast
  namespace: github-runners
spec:
  template:
    spec:
      labels:
        - x64-fast
      env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
      organization: contoso
      dockerRegistryMirror: https://mirror.gcr.io/
      nodeSelector:
        github-actions-runners-x64-fast: "true"
      tolerations:
      - key: "x64-fast"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
      dockerVolumeMounts:
      - mountPath: /var/lib/docker
        name: docker
        subPathExpr: $(POD_NAME)-docker
      - mountPath: /runner/_work
        name: work
        subPathExpr: $(POD_NAME)-work
      volumeMounts:
      - mountPath: /runner/_work
        name: work
        subPathExpr: $(POD_NAME)-work
      # - mountPath: /tmp
      #   name: tmp
      #   subPathExpr: $(POD_NAME)-tmp
      dockerEnv:
      - name: POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
      volumes:
      - hostPath:
          path: /nvme/disk
        name: docker
      - hostPath:
          path: /nvme/disk
        name: work
      # - hostPath:
      #     path: /nvme/disk
      #   name: tmp
      ephemeral: true
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: contoso-github-runners-autoscaler-x64-fast
  namespace: github-runners
spec:
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    name: contoso-github-runners-x64-fast
  minReplicas: 6
  maxReplicas: 24
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.50'
    scaleDownThreshold: '0.25'
    scaleUpFactor: '3'
    scaleDownFactor: '0.5'

Despite setting ephemeral: true, the runners don't clean up after themselves, and eventually the host EC2 machines run out of free space.

(tmp is not mounted because if we mount it we run into some weird permission errors)

What's the solution for this?

[mumoshu]

See #1695