Critical alerts not firing

[acarsercan]

Hi,

I'm failing to get prometheus rule with severity Critical to alert to Slack, if I change the severity for alertmanager to warning, everything works

Stack
kube-prometheus-stack- 0.43.2
Kubernetes 1.17

Prometheus alert manager config (helm)

    alertmanager:
      enabled: true
      config:
        route:
        group_by: ['alertname']
        group_wait: 10s
        group_interval: 10m
        repeat_interval: 3h
          receiver: 'null'
          routes:
          - match:
              alertname: Watchdog
            receiver: 'null'
          - match:
              severity: critical
            receiver: 'slack'
        receivers:
        - name: 'null'
        - name: 'slack'
          slack_configs:
          - api_url: 'https://XXXX'
            send_resolved: true

Prometheus rule to trigger

  spec:
    groups:
    - name: kubernetes-apps
      rules:
      - alert: KubePodCrashLooping
        annotations:
          description: Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container
            }}) is restarting {{ printf "%.2f" $value }} times / 5 minutes.
          runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping
          summary: Pod is crash looping.
        expr: rate(kube_pod_container_status_restarts_total{job="kube-state-metrics",
          namespace=~".*"}[5m]) * 60 * 5 > 0
        for: 15m
        labels:
          severity: critical

I have a pod which is crash looping and in prometheus UI /alerts I can see the alert is "PENDING"

Thank you

[olegmayko]

if Prometheus Alert has status "Pending". It means that Prometheus waits until for: 15m will expire. After 15m this alert will change to Firing. Hope this helps

[acarsercan]

Hi Oleg, Thank you for the reply, unfortunately 15 minutes comes and passes still no alerts. If I change the "for: 1m" then it starts firing, guessing something in the settings continuously refreshes and Prometheus never reaches 15 minutes but I'm not sure what.

…

________________________________ From: Oleg <[email protected]> Sent: 27 January 2021 20:33 To: prometheus-operator/kube-prometheus <[email protected]> Cc: Acar, Sercan <[email protected]>; Author <[email protected]> Subject: Re: [prometheus-operator/kube-prometheus] Critical alerts not firing (#896) CAUTION: This email originated from outside of the organisation. Do not click links or attachments unless you recognise the sender and know the content is safe if Prometheus Alert has status "Pending". It means that Prometheus waits until for: 15m will expire. After 15m this alert will change to Firing. Hope this helps — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprometheus-operator%2Fkube-prometheus%2Fdiscussions%2F896%23discussioncomment-315871&data=04%7C01%7C%7C03e09f50cba148affdd008d8c302d22f%7C1623e08baca149c6b57789c5bd4aa7b4%7C0%7C0%7C637473764165873774%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9La%2BoIukupJgzohyZy1bPE7LQih0REsx90t65YgI%2FYw%3D&reserved=0>, or unsubscribe<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMF66OMD2BRE2IH2DN56W3LS4B2B5ANCNFSM4WVPT3VQ&data=04%7C01%7C%7C03e09f50cba148affdd008d8c302d22f%7C1623e08baca149c6b57789c5bd4aa7b4%7C0%7C0%7C637473764165878751%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=e4mJdiNzGQD3EmNDpsixsTpO1HRc1NAzXhm0D6x5BnU%3D&reserved=0>.

[olegmayko]

I assume, that the promQL expr

rate(kube_pod_container_status_restarts_total{job="kube-state-metrics",
         namespace=~".*"}[5m]) * 60 * 5 > 0

is false at least one time during 15m. Thus, Prometheus will reset for and start the evaluation again.