Skip to content

Commit 070e26f

Browse files
authored
Add troubleshooting item (#5396)
1 parent 5151022 commit 070e26f

File tree

1 file changed

+39
-0
lines changed

1 file changed

+39
-0
lines changed

docs/send-data/kubernetes/troubleshoot-collection.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -522,6 +522,45 @@ kube-prometheus-stack:
522522

523523
where `metadata.name` is the value from Argo Application manifest.
524524

525+
### Missing metrics in dashboards
526+
527+
Dashboards can miss many metrics from the cluster and pod level. This can occur because the `kube-state-metrics` are not scraped. For example, the `prometheus-kube-state-metrics` ServiceMonitor resource does not have the label `release: prometheus`.
528+
529+
With the `prometheus` label added manually, the dashboards populate correctly and all data is pulled:
530+
531+
```yaml
532+
serviceMonitorSelector:
533+
matchLabels:
534+
monitoring: prometheus
535+
536+
kube-state-metrics:
537+
prometheus:
538+
monitor:
539+
enabled: true
540+
additionalLabels:
541+
monitoring: prometheus
542+
```
543+
544+
The `release:` label value must match the release name of the `kube-prometheus-stack` Helm deployment. The default should be `release: kube-prometheus-stack`.
545+
546+
To get the release name:
547+
548+
```text
549+
pod="$(kubectl get po -n monitoring | awk '/kube-state-metrics/{ print $1 }')"
550+
kubectl get po -n monitoring "${pod}" -o yaml | grep release
551+
```
552+
553+
Results:
554+
555+
```yaml
556+
kube-state-metrics:
557+
prometheus:
558+
monitor:
559+
enabled: true
560+
additionalLabels:
561+
release: kube-prometheus-stack
562+
```
563+
525564
### Check metrics content
526565

527566
You can print metrics on stdout of metrics collector and metrics metadata, and validate if they are correct. It may happen that metrics are ingested, but with different metadata than you expect.

0 commit comments

Comments
 (0)