External scaler goes to CrashloopBackoff
state frequently
#783
Unanswered
IsuruBoyagane15
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I have been using the add-on to scale my deployments in a AKS cluster.
I have multiple workloads to be scaled. Generally, the system works as expected.
But, I see the
external-scaler
pod goes toCrashloopBackoff
state frequently.When I checked the log in the
external-scaler
, I found this error log.{"level":"error","ts":1694510987.376597,"caller":"scaler/main.go:69","msg":"creating a queue pinger","error":"requesting the queue counts from http://10.0.7.206:9090/queue: Get \"http://10.0.7.206:9090/queue\": dial tcp 10.0.7.206:9090: i/o timeout","errorVerbose":"Get \"http://10.0.7.206:9090/queue\": dial tcp 10.0.7.206:9090: i/o timeout\nrequesting the queue counts from http://10.0.7.206:9090/queue\ngithub.com/kedacore/http-add-on/pkg/queue.GetCounts\n\t/workspace/pkg/queue/queue_rpc.go:76\nmain.fetchCounts.func1\n\t/workspace/scaler/queue_pinger.go:222\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594","stacktrace":"main.main\n\t/workspace/scaler/main.go:69\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
As I understood, the
external-scaler
could not connect to theadmin-service
of oneinterceptor
pod. (interceptor
had 3 replicas in the deployment;10.0.7.206
is the pod IP of one of the replicas)I tried roll-out restarting
external-scaler
andinterceptor
pods. No luck.During this time, the deployments that are supposed to be scaled to zero after the cool-down period (5 mins), were not scaled to zero in 5 mins.
Finally, I scaled
interceptor
deployment from 3 to 1. With that change, after some time, the setup started to work as expected.Any possible reason for this behavior?
Beta Was this translation helpful? Give feedback.
All reactions