Use kube_statefulset_replicas for KubeStatefulSetReplicasMismatch (#1026)

* Use `kube_statefulset_replicas` for `KubeStatefulSetReplicasMismatch`

According to [0] and [1] `kube_statefulset_status_replicas` returns
"the number of Pods created by the StatefulSet controller". Therefore
this alert only fires when the StatefulSet controller is able to create
the pod, but it does not get ready.

In cases where the controller is unable to create the pod (e.g. invalid
pod spec) this alert does not fire.
This is contrary to the expectation of firing when the statefulset does
not match the configured amount of replicas.

[0]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#statefulsetstatus-v1-apps
[1]: https://github.com/kubernetes/kube-state-metrics/blob/2a95d4649b2fea55799032fb9c0b571c4ba7f776/internal/store/statefulset.go#L75

* Add tests for KubeStatefulSetReplicasMismatch

Author: aeber

alerts/apps_alerts.libsonnet

@@ -121,7 +121,7 @@ local utils = import '../lib/utils.libsonnet';
               (
                 kube_statefulset_status_replicas_ready{%(prefixedNamespaceSelector)s%(kubeStateMetricsSelector)s}
                   !=
-                kube_statefulset_status_replicas{%(prefixedNamespaceSelector)s%(kubeStateMetricsSelector)s}
+                kube_statefulset_replicas{%(prefixedNamespaceSelector)s%(kubeStateMetricsSelector)s}
               ) and (
                 changes(kube_statefulset_status_replicas_updated{%(prefixedNamespaceSelector)s%(kubeStateMetricsSelector)s}[10m])
                   ==

tests/tests.yaml

@@ -1323,3 +1323,62 @@ tests:
         description: 'Cluster has overcommitted memory resource requests for Namespaces.'
         runbook_url: "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubememoryquotaovercommit"
         summary: "Cluster has overcommitted memory resource requests."
+
+# Verify KubeStatefulSetReplicasMismatch fires, when no replicas could be created
+- interval: 1m
+  name: StatefulSet replicas not created
+  input_series:
+  - series: 'kube_statefulset_replicas{cluster="kubernetes",namespace="test",job="kube-state-metrics",statefulset="sts"}'
+    values: '3x15'
+  - series: 'kube_statefulset_status_replicas{cluster="kubernetes",namespace="test",job="kube-state-metrics",statefulset="sts"}'
+    values: '0x15'
+  - series: 'kube_statefulset_status_replicas_ready{cluster="kubernetes",namespace="test",job="kube-state-metrics",statefulset="sts"}'
+    values: '0x15'
+  - series: 'kube_statefulset_status_replicas_updated{cluster="kubernetes",namespace="test",job="kube-state-metrics",statefulset="sts"}'
+    values: '0x15'
+  alert_rule_test:
+  - eval_time: 14m
+    alertname: KubeStatefulSetReplicasMismatch
+  - eval_time: 15m
+    alertname: KubeStatefulSetReplicasMismatch
+    exp_alerts:
+    - exp_labels:
+        severity: "warning"
+        job: "kube-state-metrics"
+        cluster: "kubernetes"
+        namespace: "test"
+        statefulset: "sts"
+      exp_annotations:
+        description: "StatefulSet test/sts has not matched the expected number of replicas for longer than 15 minutes."
+        runbook_url: "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetreplicasmismatch"
+        summary: "StatefulSet has not matched the expected number of replicas."
+
+# Verify KubeStatefulSetReplicasMismatch fires, when replicas could be created but are not ready
+- interval: 1m
+  name: StatefulSet replicas created but not ready
+  input_series:
+  - series: 'kube_statefulset_replicas{cluster="kubernetes",namespace="test",job="kube-state-metrics",statefulset="sts"}'
+    values: '3x15'
+  - series: 'kube_statefulset_status_replicas{cluster="kubernetes",namespace="test",job="kube-state-metrics",statefulset="sts"}'
+    values: '3x15'
+  - series: 'kube_statefulset_status_replicas_ready{cluster="kubernetes",namespace="test",job="kube-state-metrics",statefulset="sts"}'
+    values: '0x15'
+  - series: 'kube_statefulset_status_replicas_updated{cluster="kubernetes",namespace="test",job="kube-state-metrics",statefulset="sts"}'
+    values: '0x15'
+  alert_rule_test:
+  - eval_time: 14m
+    alertname: KubeStatefulSetReplicasMismatch
+  - eval_time: 15m
+    alertname: KubeStatefulSetReplicasMismatch
+    exp_alerts:
+    - exp_labels:
+        severity: "warning"
+        job: "kube-state-metrics"
+        cluster: "kubernetes"
+        namespace: "test"
+        statefulset: "sts"
+      exp_annotations:
+        description: "StatefulSet test/sts has not matched the expected number of replicas for longer than 15 minutes."
+        runbook_url: "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubestatefulsetreplicasmismatch"
+        summary: "StatefulSet has not matched the expected number of replicas."
+