Problem running manager "error": "leader election lost"

[Edarjak]

Hi community,

Maybe someone ever faced the same issue as me?

actions-runner-controller pod logs:

E0427 12:19:29.308086       1 leaderelection.go:330] error retrieving resource lock actions-runner-system/actions-runner-controller: Get "https://10.43.0.1:443/apis/coordination.k8s.io/v1/namespaces/actions-runner-system/leases/actions-runner-controller": context deadline exceeded
I0427 12:19:29.308753       1 leaderelection.go:283] failed to renew lease actions-runner-system/actions-runner-controller: timed out waiting for the condition
2023-04-27T12:19:29Z	INFO	Stopping and waiting for non leader election runnables
2023-04-27T12:19:29Z	INFO	Stopping and waiting for leader election runnables
2023-04-27T12:19:29Z	INFO	Stopping and waiting for caches
2023-04-27T12:19:29Z	INFO	Stopping and waiting for webhooks
2023-04-27T12:19:29Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim"}
2023-04-27T12:19:29Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet"}
2023-04-27T12:19:29Z	ERROR	problem running manager	{"error": "leader election lost"}
main.main
	./main.go:397
runtime.main
	runtime/proc.go:250
2023-04-27T12:19:29Z	INFO	Shutdown signal received, waiting for all workers to finish	{"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod"}

Would be very thankful for any advice!

Br,

[kevholmes]

It looks like the pod is unable to speak to your k8s API? Check your VPC firewall or NetworkPolicy resources for a rule that might be blocking that kind of traffic?

leaderelection.go:330] error retrieving resource lock actions-runner-system/actions-runner-controller: Get "https://10.43.0.1:443/apis/coordination.k8s.io/v1/namespaces/actions-runner-system/leases/actions-runner-controller": context deadline exceeded
leaderelection.go:283] failed to renew lease actions-runner-system/actions-runner-controller: timed out waiting for the condition

[kevholmes]

@Edarjak when the controller gets in this state where it's unable to speak to K8s API - can you drop into the ARC controller pod directly or use kubectl debug ... to attach a debug container to the controller pod and try to run a curl request against the endpoint that it's timing out with, eg curl -k -I https://10.43.0.1:443/apis/coordination.k8s.io/v1/namespaces/actions-runner-system/leases/actions-runner-controller and see if you get a timeout? Also might be worth seeing if other unrelated pods are having this connectivity issue, also.

Do you have any metrics on k8s API request latency? I could see if the k8s cluster is heavily burdened we might be timing out on the lease endpoint's API requests. Do you have a high number of objects in the k8s API - that might also cause enough latency to start timing out here in my experience.

[Edarjak]

Thanks for your input @kevholmes!

I tried all recommendations:

1. dnstools container in the ARC pod:

### Container started, was working for 15-20 min

dnstools# curl -k -I https://10.43.0.1:443/apis/coordination.k8s.io/v1/namespaces/actions-runner-system/leases/actions-runner-controller
HTTP/1.1 401 Unauthorized
Audit-Id: df4ca858-7918-4873-90cf-b2283d3bc5bf
Cache-Control: no-cache, private
Content-Type: application/json
Date: Tue, 02 May 2023 11:28:06 GMT
Content-Length: 157

### Crashed

dnstools# curl -k -I https://10.43.0.1:443/apis/coordination.k8s.io/v1/namespaces/actions-runner-system/leases/actions-runner-controller

When it crashed, it couldn't output anything because of the time out.

Only the manager container failed:

actions-runner-controller-798b5fdf46-4699f:
	manager
	kube-rbac-proxy

Manager logs:

2023-05-02T11:34:49Z	ERROR	Failed to get API Group-Resources	{"error": "Get \"https://10.43.0.1:443/api?timeout=32s\": dial tcp 10.43.0.1:443: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/cluster.New
	sigs.k8s.io/[email protected]/pkg/cluster/cluster.go:161
sigs.k8s.io/controller-runtime/pkg/manager.New
	sigs.k8s.io/[email protected]/pkg/manager/manager.go:359
main.main
	./main.go:173
runtime.main
	runtime/proc.go:250
2023-05-02T11:34:49Z	ERROR	unable to start manager	{"error": "Get \"https://10.43.0.1:443/api?timeout=32s\": dial tcp 10.43.0.1:443: i/o timeout"}
main.main
	./main.go:188
runtime.main

Events:

26m         Normal    Created            pod/actions-runner-controller-798b5fdf46-4699f    Created container debugger-gfzp5
26m         Normal    Started            pod/actions-runner-controller-798b5fdf46-4699f    Started container debugger-gfzp5
16m         Normal    Pulling            pod/actions-runner-controller-798b5fdf46-4699f    Pulling image "my-repo.artifactory.host.com/infoblox/dnstools:latest"
16m         Normal    Pulled             pod/actions-runner-controller-798b5fdf46-4699f    Successfully pulled image "my-repo.artifactory.host.com/infoblox/dnstools:latest" in 118.999876ms (119.018756ms including waiting)
16m         Normal    Created            pod/actions-runner-controller-798b5fdf46-4699f    Created container debugger-ldqr9
16m         Normal    Started            pod/actions-runner-controller-798b5fdf46-4699f    Started container debugger-ldqr9
25s         Warning   BackOff            pod/actions-runner-controller-798b5fdf46-4699f    Back-off restarting failed container manager in pod actions-runner-controller-798b5fdf46-4699f_actions-runner-system(0147fc77-474f-4f9e-8095-f9a4e4f8cfe8)
33m         Warning   BackOff            pod/actions-runner-controller-798b5fdf46-xjkz5    Back-off restarting failed container manager in pod actions-runner-controller-798b5fdf46-xjkz5_actions-runner-system(58f9b3db-970e-4a9b-9055-c739a0c8714a)
30m         Normal    SuccessfulCreate   replicaset/actions-runner-controller-798b5fdf46   Created pod: actions-runner-controller-798b5fdf46-4699f
30m         Normal    LeaderElection     lease/actions-runner-controller                   actions-runner-controller-798b5fdf46-4699f_f2ce5f00-a9fa-4bbe-b6bf-134d28071e73 became leader

2. dnstools container in the Runner pod:

dnstools# curl -k -I https://10.43.0.1:443

dnstools# nc -v 10.43.0.1 443

Time out as well.

For me, it's a bit unclear what is causing that issue. It's a test cluster with only two active runner pods, so shouldn't cause latency.

Could you recommend what and how should be checked first in this case?

Br,

[kevholmes]

@Edarjak the fact that it works for 15-20 minutes but then just completely looses connectivity is very strange. It sounds to me like there's some kind of config issue or hardware problem with your k8s cluster? Can you look at logs for kube-proxy or on the nodes themselves to see if something else is going on here? Are you using a custom CNI provider like Calico, Weave or Cilium?

[kevholmes]

Any daemonsets that are modifying your nodes?

[Edarjak]

Thanks for your reply @kevholmes! It might be an issue with the K3S cluster. I'm using Flannel CNI. Not sure, but I think daemonsets are not in use.

[Edarjak]

Internal Chef cookbook was causing that issue