Kube-state-metrics pod is getting restarted without any logs

[nirmal20081028]

Kube-state-metrics pod is getting restarted without any events/logs.
Below checks are validated:
-- pod events
-- pod logs
-- CPU and memory history.

Could you please help what could the reasons.

helm.sh/chart = kube-state-metrics-5.27.0
pod output:
prod-kube-state-metrics-78b66b5600-gm9mv 1/1 Running 13 (3d8h ago) 27d

[richabanker]

Hi, could you share more details about your cluster? How many instances / pods do you have or any other logs would help

[richabanker]

ping @nirmal20081028

[andreas-p]

I have the same issue on a 1.33 cluster, with 2.16 and 2.15. There are only endpoint deprecation and startup messages in the logs, the container exits with error code 2. So far, I haven't seen any correlation to events on the cluster (some 50 pods).

[alexifsp]

I observed also some issues with the kube-state-metrics pod getting OOMKilled:

helm.sh/chart=kube-state-metrics-6.1.0

In my case I updated resources:

    Limits:
      cpu:                250m
      ephemeral-storage:  100Mi
      memory:             1Gi
    Requests:
      cpu:                100m
      ephemeral-storage:  50Mi
      memory:             256Mi
    Liveness:             http-get http://:8080/livez delay=5s timeout=5s period=10s #success=1 #failure=3
    Readiness:            http-get http://:8081/readyz delay=5s timeout=5s period=10s #success=1 #failure=3
    Environment:          <none>

@nirmal20081028 Could you describe your pod too? Just to eliminate this from possible causes?

[andreas-p]

My issue is definitively not OOM, which would be error code 137 and show up in the host kern.log as well.

[mrueg]

Please provide a full log output and information on your specific configuration (e.g. if you use customresourcestate), otherwise it's very difficult to help.

Also, were the /livez /healthz /metrics endpoints available before it crashed?

[andreas-p]

The pods crash after hours or days of uptime, with no logging at all. The pods are generated from Helm Chart 27.28.0, with resources defined as cpu req=10m, limit=30m, mem req=40Mi, limit=80Mi. I've never observed resource problems (currently top pod: cpu=1m, mem=53Mi)

[assu-2000]

hi all, I've been following this thread and wanted to summarize the situation to help focus the debugging efforts.

summary of the Issue

It appears we might be discussing 2 separate issues in this thread:

1. Unexpected Exit with Code 2: as reported by @nirmal20081028 and @andreas-p , the kube-state-metrics container stops and exits with code 2. this happens silently after hours or days, without any error logs or OOMKilled events. This seems to be the core, unresolved issue.
2. OOMKilled Restarts: as reported by @alexifsp , the pod is being terminated due to memory limits (OOMKilled, which results in exit code 137). this is a resource config issue, which was resolved by increasing the memory limit.

To move forward, it's crucial to focus on the exit code 2 problem, as its cause is not imediately clear
@mrueg have asked for key information that is still needed.

To help diagnose this, can @nirmal20081028 and @andreas-p provide the following?

1. full logs from the Previous Container -> immediately after a crash, please try to run kubectl logs --previous <your-kube-state-metrics-pod-name> -n <namespace>; even if it seems empty, please consider sharing the full output here
2. try to enable Verbose Logging by adding the --v=4 argument to the k-s-m container spec in your Deployment. this setup can generate much more detailed logs and will likely capture what happens right before the next crash.

   # example snippet for your depl. manifest
   spec:
     containers:
     - name: kube-state-metrics
       args:
       - --v=4 
       # ... other args

3. full configuration: please share your Helm values.yaml file or the full arguments passed to the KSM container
4. Liveness/Readiness Probe Status: can you confirm if the /livez and /readyz endpoints were responding correctly before the crash? a failing liveness probe would also cause a restart, but according to the docs this is usually logged as an event.

I think these information will give the necessary context to identify the root cause. Thank you!

[andreas-p]

Running with --v=4 now, waiting for the crash.

[dgrisonnet]

/triage accepted

[dgrisonnet]

@andreas-p friendly ping for the information asked previously by @assu-2000