PrometheusNotIngestingSamples rule alerting for type "histogram"

[frederiksf]

Since we move from version 0.10 to version 0.12 (prometheus release 2.41.0) the following alerting rule fires

name: PrometheusNotIngestingSamples
expr: rate(prometheus_tsdb_head_samples_appended_total{job="prometheus-k8s",namespace="monitoring"}[5m]) <= 0
for: 10m
labels:
severity: warning
annotations:
description: Prometheus {{$labels.namespace}}/{{$labels.pod}} is not ingesting samples.
summary: Prometheus is not ingesting samples.

After some research i see that type floating has values > 0 but type histogram is the one which fires the alarm.

What i do not understand is that i have histogram metrics which i can scrape successfull. for example:

In the previous version of prometheus (2.32.1) the was only one metric without the difference between type=count or type=histogram

Does anybody understand what is happening here? Thank you!

[VerosK]

Wasn't this issue resolved in #2095 ?

[palmobar]

@VerosK this is the issue #2095 x| @frederiksf did you manage to resolve it?

[tamsky]

Author opened both a discussion (#2095) and an issue (#2096).

Issue appears unresolved, I'm seeing this behavior as well.

Code was added here to create separate {type="float|histogram"} labelled metrics:

• prometheus/prometheus@d0a6488#diff-c3ae701c35bb1de34870207029d1fb3390613fbba65e6f2f9bffe059bff89001R765-R766

Current rule @current{main} ::

• kube-prometheus/manifests/prometheus-prometheusRule.yaml

  Line 111 in 858a2a5

  rate(prometheus_tsdb_head_samples_appended_total{job="prometheus-k8s",namespace="monitoring"}[5m]) <= 0

This rule probably should be updated to alert only on the old behavior by adding the type="float" to the labelset.