Standby Replicas without lease use lots of CPU #174
Description
I'm using the Helm Chart to deploy openshift-routes 0.8.0 0.7.1 with two replicas.
Without any Routes to handle, the active replica uses 1 mCPU (renewing leases every 2 seconds), but the inactive one can quickly start hogging all available CPU cores with no CPU limits set:
Interestingly enough, it looks like this doesn't happen on every cluster of ours, but not sure yet why.
Snippet from log of openshift-routes:
I0411 10:35:26.527477 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
I0411 10:35:31.630770 1 leaderelection.go:456] lock is held by cert-manager-openshift-routes-54c4fc89cd-h7rh4_2c94fd85-aa32-4a45-961d-7fb727fe584f and has not yet expired
I0411 10:35:31.687501 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
I0411 10:35:35.338385 1 request.go:632] Waited for 186.705849ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/coordination.k8s.io/v1/namespaces/cert-man
I0411 10:35:40.009979 1 leaderelection.go:456] lock is held by cert-manager-openshift-routes-54c4fc89cd-h7rh4_2c94fd85-aa32-4a45-961d-7fb727fe584f and has not yet expired
I0411 10:35:40.120232 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
I0411 10:35:46.303026 1 leaderelection.go:456] lock is held by cert-manager-openshift-routes-54c4fc89cd-h7rh4_2c94fd85-aa32-4a45-961d-7fb727fe584f and has not yet expired
I0411 10:35:46.473547 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
I0411 10:35:53.073197 1 leaderelection.go:456] lock is held by cert-manager-openshift-routes-54c4fc89cd-h7rh4_2c94fd85-aa32-4a45-961d-7fb727fe584f and has not yet expired
I0411 10:35:53.178306 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
I0411 10:35:58.039708 1 leaderelection.go:456] lock is held by cert-manager-openshift-routes-54c4fc89cd-h7rh4_2c94fd85-aa32-4a45-961d-7fb727fe584f and has not yet expired
I0411 10:35:58.765609 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
I0411 10:36:04.821075 1 request.go:632] Waited for 113.586318ms due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/coordination.k8s.io/v1/namespaces/cert-man
[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
> goroutine 678 [running]:
> runtime/debug.Stack()
> runtime/debug/stack.go:26 +0x5e
> sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
> sigs.k8s.io/[email protected]/pkg/log/log.go:60 +0xcd
> sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc0002baf00, 0x1)
> sigs.k8s.io/[email protected]/pkg/log/deleg.go:111 +0x32
> github.com/go-logr/logr.Logger.Info({{0x1f8dd40?, 0xc0002baf00?}, 0x1ca87a0?}, {0x1cb00b6, 0x14}, {0xc0001c2d40, 0x4, 0x4})
> github.com/go-logr/[email protected]/logr.go:276 +0x6e
> sigs.k8s.io/controller-runtime/pkg/healthz.(*Handler).serveAggregated(0xc00009c460, {0x1f79698, 0xc0008f2000}, 0xc00089e3c0)
> sigs.k8s.io/[email protected]/pkg/healthz/healthz.go:60 +0x3b0
> sigs.k8s.io/controller-runtime/pkg/healthz.(*Handler).ServeHTTP(0xc00009c460, {0x1f79698, 0xc0008f2000}, 0xc00089e3c0)
> sigs.k8s.io/[email protected]/pkg/healthz/healthz.go:148 +0x8a
> sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).addHealthProbeServer.StripPrefix.func1({0x1f79698, 0xc0008f2000}, 0xc000b12280)
> net/http/server.go:2282 +0x262
> net/http.HandlerFunc.ServeHTTP(0xc000156d20?, {0x1f79698?, 0xc0008f2000?}, 0xc?)
> net/http/server.go:2220 +0x29
> net/http.(*ServeMux).ServeHTTP(0x46e1f9?, {0x1f79698, 0xc0008f2000}, 0xc000b12280)
> net/http/server.go:2747 +0x1ca
> net/http.serverHandler.ServeHTTP({0xc0005d86f0?}, {0x1f79698?, 0xc0008f2000?}, 0x6?)
> net/http/server.go:3210 +0x8e
> net/http.(*conn).serve(0xc0008a01b0, {0x1f88680, 0xc000561740})
> net/http/server.go:2092 +0x5d0
> created by net/http.(*Server).Serve in goroutine 81
> net/http/server.go:3360 +0x485
I0411 10:36:30.274573 1 leaderelection.go:456] lock is held by cert-manager-openshift-routes-54c4fc89cd-h7rh4_2c94fd85-aa32-4a45-961d-7fb727fe584f and has not yet expired
I0411 10:36:31.384188 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
I0411 10:36:37.378234 1 leaderelection.go:456] lock is held by cert-manager-openshift-routes-54c4fc89cd-h7rh4_2c94fd85-aa32-4a45-961d-7fb727fe584f and has not yet expired
I0411 10:36:37.435497 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
I0411 10:36:42.322390 1 leaderelection.go:456] lock is held by cert-manager-openshift-routes-54c4fc89cd-h7rh4_2c94fd85-aa32-4a45-961d-7fb727fe584f and has not yet expired
I0411 10:36:42.345716 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
I0411 10:36:47.956576 1 leaderelection.go:456] lock is held by cert-manager-openshift-routes-54c4fc89cd-h7rh4_2c94fd85-aa32-4a45-961d-7fb727fe584f and has not yet expired
I0411 10:36:48.092046 1 leaderelection.go:263] failed to acquire lease cert-manager/cert-manager-openshift-routes
Edit:
Correction, we're still running 0.7.1 - will test with 0.8.0 and share my findings.