Having trouble running Kind action

[planetf1]

Hi,
I've hit some job resource limitations with github actions using github runners for an open project. The 6 hours/7GB is too restrictive for some intensive tests :-(

Whilst one approach is to deploy to an external k8s cluster and just manage the deployment/data capture from the runner, I wondered about using a custom runner instead, just to see what it would take

So I've just deployed the action-runner-controller to an Openshift 4.11 cluster. I added the 'privileged' scc, and it's able to create runners as I want. I can customize memory, cpu requirements etc in the CR.

However whilst the runners go Idle, then active, and start executing my actinos, I find that my 'kind' action

      - uses: helm/kind-action@v1
        name: setup kind
        with:
          cluster_name: cts
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

just hangs/times out.

I initially tried with a simple CR, then added additional resources, then specific docker config, but still get the issue.

Any ideas?

 apiVersion: actions.summerwind.dev/v1alpha1
 kind: RunnerDeployment
 metadata:
   namespace: runner
   name: egeria-runner
 spec:
   replicas: 4
   template:
     spec:
       repository: planetf1/cts
       resources:
         limits:
           cpu: "4000m"
           memory: "12Gi"
         requests:
           cpu: "500m"
           memory: "6Gi"
       dockerEnabled: true
       dockerdWithinRunnerContainer: false
       dockerdContainerResources:
         limits:
           cpu: "4000m"
           memory: "12Gi"
         requests:
           cpu: "500m"
           memory: "6Gi"

From the action log, it would appear to be a docker issue:

Run helm/kind-action@v1
[35](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:36)
Installing kind...
[36](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:37)
Adding kind directory to PATH...
[37](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:38)
Installing kubectl...
[38](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:39)
Adding kubectl directory to PATH...
[39](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:40)
kind v0.16.0 go1.19.1 linux/amd64
[40](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:41)
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.12", GitCommit:"c6939792865ef0f70f92006081690d77411c8ed5", GitTreeState:"clean", BuildDate:"2022-09-21T12:20:29Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
[41](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:42)
Creating kind cluster...
[42](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:43)
Creating cluster "cts" ...
[43](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:44)
 • Ensuring node image (kindest/node:v1.25.2) 🖼  ...
[44](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:45)
 ✓ Ensuring node image (kindest/node:v1.25.2) 🖼
[45](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:46)
 • Preparing nodes 📦   ...
[46](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:47)
 ✓ Preparing nodes 📦 
[47](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:48)
 • Writing configuration 📜  ...
[48](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:49)
 ✓ Writing configuration 📜
[49](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:50)
 • Starting control-plane 🕹️  ...
[50](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:51)
 ✗ Starting control-plane 🕹️
[51](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:52)
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged cts-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
[52](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:53)

It appears to be a timeout sometime after:

[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[116](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:117)
I1215 11:59:23.234910     251 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
[117](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:118)
I1215 11:59:23.234933     251 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy
[118](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:119)
I1215 11:59:23.235744     251 loader.go:374] Config loaded from file:  /etc/kubernetes/admin.conf
[119](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:120)
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[120](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:121)
I1215 11:59:23.239537     251 round_trippers.go:553] GET https://cts-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[121](https://github.com/planetf1/cts/actions/runs/3703858356/jobs/6275800102#step:5:122)
I1215 11:59:23.740946     251 round_trippers.go:553] GET https://cts-control-plane:6443/healthz?timeout=10s  in 0 milliseconds

I realise this is kind-a a kind question, but the same action is working fine on github hosted runners

I haven't yet found more detail on the pods running the runners yet.. still looking

[planetf1]

The docker container logs have some errors:

➜  workflows git:(main) kubectl logs egeria-runner-kqbtb-mh8tv -c docker | grep -y 'level=error'
time="2022-12-15T11:57:54.041930985Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
time="2022-12-15T11:57:54.061563670Z" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
time="2022-12-15T11:57:54.061730374Z" level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fuse-overlayfs
time="2022-12-15T11:57:54.068316598Z" level=error msg="AUFS was not found in /proc/filesystems" storage-driver=aufs
time="2022-12-15T11:57:54.072509869Z" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay
time="2022-12-15T11:57:54.072540052Z" level=error msg="Failed to built-in GetDriver graph devicemapper /var/lib/docker"
time="2022-12-15T13:24:13.586135903Z" level=error msg="Could not add route to IPv6 network fc00:f853:ccd:e793::1/64 via device br-6e327cd26ee3: network is down"

[planetf1]

I also get the same with:

       image: summerwind/actions-runner-dind
       dockerdWithinRunnerContainer: true