Docker Container is Erroring Out

[chadsaun]

I had been running ARC in my GCP k8s cluster for months and then it just stopped working. The pod that runs the two containers runner and docker is in a status of Error for container docker. I've also tried uninstalling the Helm chart (because that's how I installed it) and then re-installing the Helm chart and still get the same error.

Here's how I installed it: helm upgrade --install --namespace actions-runner-system --create-namespace --wait actions-runner-controller actions-runner-controller/actions-runner-controller --set syncPeriod=1m --set authSecret.create=true --set authSecret.github_token=*****

Here's what the logs show for the docker container.

failed to start daemon: Error initializing network controller: error obtaining controller instance: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain: (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables): RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1

Here's more of the logs sequence leading up to the error.

ERROR 2024-01-19T18:10:52.067167186Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.067006533Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
ERROR 2024-01-19T18:10:52.067172522Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.067026142Z" level=info msg="skipping tracing processor initialization (no tracing plugin)" error="no OpenTelemetry endpoint: skip plugin"
ERROR 2024-01-19T18:10:52.068252370Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.068153303Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
ERROR 2024-01-19T18:10:52.068657927Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.068563469Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
ERROR 2024-01-19T18:10:52.069104054Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.068979595Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
ERROR 2024-01-19T18:10:52.069128731Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.069045016Z" level=info msg="containerd successfully booted in 0.097493s"
ERROR 2024-01-19T18:10:52.151773603Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.151608115Z" level=info msg="Loading containers: start."
ERROR 2024-01-19T18:10:52.212289979Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.210214153Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
ERROR 2024-01-19T18:10:52.212361540Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.211245303Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
ERROR 2024-01-19T18:10:52.212370607Z [resource.labels.containerName: docker] time="2024-01-19T18:10:52.211294761Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
ERROR 2024-01-19T18:10:53.212011606Z [resource.labels.containerName: docker] failed to start daemon: Error initializing network controller: error obtaining controller instance: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain: (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables): RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1
ERROR 2024-01-19T18:10:53.212052853Z [resource.labels.containerName: docker] (exit status 4))

Any ideas as to what is causing this error?