How to know what's wrong when a runner pod crash

[pckhoi]

I started a job which run for 1 hour until the runner crash and I never know what happened:

Here's the logs from the runner pod (all from docker container) right before it crashed:

time="2022-10-15T05:02:51.380982878Z" level=info msg="Processing signal 'terminated'"
time="2022-10-15T05:02:59.671195207Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
time="2022-10-15T05:02:59.671485609Z" level=info msg="Daemon shutdown complete"
time="2022-10-15T05:02:59.671512006Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2022-10-15T05:02:59.671533093Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2022-10-15T05:03:00.672032774Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}. Err :connection error: desc = \\"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\\". Reconnecting..." module=grpc

And here's the logs from the controller pod at that time:

2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "7acd58ef-bbfc-404b-bb16-7cbe22b6bae8", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "7acd58ef-bbfc-404b-bb16-7cbe22b6bae8", "allowed": true}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "2a044f3f-85f8-4135-a8d2-d814ff97f813", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "2a044f3f-85f8-4135-a8d2-d814ff97f813", "allowed": true}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "3c98bef5-a673-4441-9e51-4142ca352ad2", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-15T05:03:10Z    INFO    runner-resource validate resource to be created {"name": "cdi-runner-6qc22-6fpmx"}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "3c98bef5-a673-4441-9e51-4142ca352ad2", "allowed": true}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "efe13886-416f-4649-86d7-953195927cce", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-15T05:03:10Z    INFO    runner-resource validate resource to be updated {"name": "cdi-runner-6qc22-5bv7l"}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "efe13886-416f-4649-86d7-953195927cce", "allowed": true}
2022-10-15T05:03:10Z    DEBUG   actions-runner-controller.runnerreplicaset      Created replica(s)      {"runnerreplicaset": "cdi-runner/cdi-runner-6qc22", "lastSyncTime": null, "effectiveTime": "<nil>", "templateHashDesired": "74496784b7", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2022-10-15T05:03:10Z    DEBUG   actions-runner-controller.runnerreplicaset      Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "cdi-runner/cdi-runner-6qc22", "owner": "cdi-runner/cdi-runner-6qc22-6fpmx", "pods": null}
2022-10-15T05:03:10Z    DEBUG   actions-runner-controller.runnerreplicaset      Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "cdi-runner/cdi-runner-6qc22", "owner": "cdi-runner/cdi-runner-6qc22-6fpmx", "pods": null}
2022-10-15T05:03:10Z    INFO    actions-runner-controller.runner        Removed finalizer       {"runner": "cdi-runner/cdi-runner-6qc22-5bv7l"}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "UID": "4a3f1c09-2c82-4b04-95d5-295e44f471c0", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/mutate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "4a3f1c09-2c82-4b04-95d5-295e44f471c0", "allowed": true}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "UID": "49bac983-026e-42e8-ac9d-9b413f21625d", "kind": "actions.summerwind.dev/v1alpha1, Kind=Runner", "resource": {"group":"actions.summerwind.dev","version":"v1alpha1","resource":"runners"}}
2022-10-15T05:03:10Z    INFO    runner-resource validate resource to be updated {"name": "cdi-runner-6qc22-6fpmx"}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/validate-actions-summerwind-dev-v1alpha1-runner", "code": 200, "reason": "", "UID": "49bac983-026e-42e8-ac9d-9b413f21625d", "allowed": true}
2022-10-15T05:03:10Z    DEBUG   actions-runner-controller.runnerreplicaset      Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "cdi-runner/cdi-runner-6qc22", "owner": "cdi-runner/cdi-runner-6qc22-6fpmx", "pods": null}
2022-10-15T05:03:10Z    INFO    actions-runner-controller.runnerpod     Runner pod has been stopped with a successful status.   {"runnerpod": "cdi-runner/cdi-runner-6qc22-5bv7l"}
2022-10-15T05:03:10Z    INFO    actions-runner-controller.runner        Updated registration token      {"runner": "cdi-runner-6qc22-6fpmx", "repository": "ppact/processing"}
2022-10-15T05:03:10Z    DEBUG   events  Normal  {"object": {"kind":"Runner","namespace":"cdi-runner","name":"cdi-runner-6qc22-6fpmx","uid":"de0f30e6-be16-4700-b0f6-ae94e12a851d","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"6391478"}, "reason": "RegistrationTokenUpdated", "message": "Successfully update registration token"}
2022-10-15T05:03:10Z    DEBUG   actions-runner-controller.runnerreplicaset      Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "cdi-runner/cdi-runner-6qc22", "owner": "cdi-runner/cdi-runner-6qc22-6fpmx", "pods": null}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/mutate-runner-set-pod", "UID": "2580eab7-1af2-47ad-aa99-f716d98dd74f", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/mutate-runner-set-pod", "code": 200, "reason": "", "UID": "2580eab7-1af2-47ad-aa99-f716d98dd74f", "allowed": true}
2022-10-15T05:03:10Z    INFO    actions-runner-controller.runner        Created runner pod      {"runner": "cdi-runner/cdi-runner-6qc22-6fpmx", "repository": "ppact/processing"}
2022-10-15T05:03:10Z    DEBUG   events  Normal  {"object": {"kind":"Runner","namespace":"cdi-runner","name":"cdi-runner-6qc22-6fpmx","uid":"de0f30e6-be16-4700-b0f6-ae94e12a851d","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"6391483"}, "reason": "PodCreated", "message": "Created pod 'cdi-runner-6qc22-6fpmx'"}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     received request        {"webhook": "/mutate-runner-set-pod", "UID": "79e71f51-bda8-40fe-a872-c06085672bb2", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2022-10-15T05:03:10Z    DEBUG   controller-runtime.webhook.webhooks     wrote response  {"webhook": "/mutate-runner-set-pod", "code": 200, "reason": "", "UID": "79e71f51-bda8-40fe-a872-c06085672bb2", "allowed": true}
2022-10-15T05:03:10Z    INFO    actions-runner-controller.runner        Failed to create pod due to AlreadyExists error. Probably this pod has been already created in previous reconcilation but is still not in the informer cache. Will retry on pod created. If it doesn't repeat, there's no problem    {"runner": "cdi-runner/cdi-runner-6qc22-6fpmx"}
2022-10-15T05:08:12Z    DEBUG   actions-runner-controller.runner        Runner appears to have been registered and running.     {"runner": "cdi-runner/cdi-runner-6qc22-6fpmx", "podCreationTimestamp": "2022-10-15 05:03:10 +0000 UTC"}

How I deployed ARC

• Deployed with kubectl create/apply/replace.
• ARC version 0.26.0.
• cert-manager version v1.8.2
• runner deployment spec:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: cdi-runner
  namespace: cdi-runner
spec:
  replicas: 1
  template:
    spec:
      repository: ppact/processing
      image: ghrunner
      labels:
        - cdi-runner
      env: []
      resources:
        limits:
          cpu: "4.0"
          memory: "8Gi"
        requests:
          cpu: "2.0"
          memory: "4Gi"
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: cdi-runner-data
      volumeMounts:
        - mountPath: /runner/_work/data
          name: data

[Diontsoumas]

Is it possible to add logs yourself, or enable a debug flag so you can have a better understanding about what is happening when the command runs?

I would start by making sure that it wasn't an OOM error.