Github service container do not work with containerMode: kubernetes

[Brenner87]

Hello. I've started playing with containerMode: kubernetes. Everything works as expected while I'm not using GitHub services.
In pod description I can see error during service container start:

Warning  Failed   4s    kubelet  Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "": executable file not found in $PATH: unknown

Currently I have 0.21.1 helm release installed.
Here is my deploymen and autoscaler:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: k8s-resources-github-runner-deployment
  namespace: github-actions-runners
spec:
  template:
    metadata:
      labels:
        vault-injection: "true"
    spec:
      serviceAccountName: githubrunner
      containerMode: kubernetes
      workVolumeClaimTemplate:
        storageClassName: synology-csi-normal
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
      organization: myfancyorganization
      labels:
        - k8s-resources
        - github-action-runner-pod-k8s-resources
        - on-prem-k8s-resources
        - development-cluster-k8s-resources
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: k8s-resources-github-runner-deployment-autoscaler
  namespace: github-actions-runners
spec:
  scaleTargetRef:
    kind: RunnerDeployment
    name: k8s-resources-github-runner-deployment
  minReplicas: 1
  maxReplicas: 10
  metrics:
    #Documentation: https://github.com/actions/actions-runner-controller/blob/master/docs/detailed-docs.md
    - type: PercentageRunnersBusy
      scaleUpThreshold: "0.75" # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up
      scaleDownThreshold: "0.5" # The percentage of busy runners at which the number of desired runners are re-evaluated to scale down
      scaleUpAdjustment: 2 # The scale up runner count added to desired count
      scaleDownAdjustment: 1 # The scale down runner count subtracted from the desired count

Here is service account part:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: github-actions-runners
  name: githubrunner-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["get", "create"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "watch",]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: githubrunner-rb
  namespace: github-actions-runners
subjects:
- kind: ServiceAccount
  name: githubrunner
  namespace: github-actions-runners
roleRef:
  kind: Role
  name: githubrunner-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: githubrunner

And here is workflow I'm trying to execute:

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]
  workflow_dispatch:
jobs:
  step2:
    runs-on: [on-prem-k8s-resources]
    services:
      postgres:
        image: postgres:latest
    container:
      image: <path-to-internal-registry>/ops-pipeline:6.0.0
    steps:
      - name: Run a multi-line script
        run: |
          whoami
          hostname
          sleep 200

In workflow, I'm getting following error:

And here is k describe pod k8s-resources-github-runner-deployment-ggxwz-j72hh-workflow output:

  Type     Reason   Age   From     Message
  ----     ------   ----  ----     -------
  Normal   Pulling  105s  kubelet  Pulling image "docker.fntgrp.com/fntsoftware/it-services/tools/ops-pipeline:6.0.0"
  Normal   Pulled   65s   kubelet  Successfully pulled image "docker.fntgrp.com/fntsoftware/it-services/tools/ops-pipeline:6.0.0" in 39.268879206s
  Normal   Created  65s   kubelet  Created container job
  Normal   Started  64s   kubelet  Started container job
  Normal   Pulling  64s   kubelet  Pulling image "postgres:latest"
  Normal   Pulled   25s   kubelet  Successfully pulled image "postgres:latest" in 39.38886194s
  Normal   Created  5s    kubelet  Created container postgres
  Warning  Failed   4s    kubelet  Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "": executable file not found in $PATH: unknown

There is really no runc on actions runner(using default one).

What am I doing wrong? Can someone assist?

[jdrinkwater-literati]

I also hit this issue, and can add even more context.

Also found in the describe of the job pod, you can see an empty Command: field for the container.

postgres:
    Container ID:  containerd://ccee788c44117d6aa7256aa584ea22ffb3e118dc51c17fb3501bb33194d8ed15
    Image:         postgres:latest
    Image ID:      docker.io/library/postgres@sha256:6cc97262444f1c45171081bc5a1d4c28b883ea46a6e0d1a45a8eac4a7f4767ab
    Port:          <none>
    Host Port:     <none>
    Command:

    State:          Terminated
      Reason:       StartError
      Message:      failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "": executable file not found in $PATH: unknown
      Exit Code:    128

This same container when pulled and inspected on my local machine has a non-empty entrypoint and cmd to start postgres (unsurprisingly)

It would seem that somewhere along the way, the startup command for the container is being replaced with an empty string, perhaps when it should have been <none> like the Port and Host Port fields above it, or being omitted completely (unsure which is the correct way to do that)