Skip to content

fix: Look for increases in counter for missing_pods Prometheus alert rule #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mvlassis merged 7 commits into from
Feb 12, 2026
Merged

fix: Look for increases in counter for missing_pods Prometheus alert rule #310

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
98f07c5
chore: Look for increases in counter for Prometheus alert rule
mvlassis Feb 5, 2026
bb32fe5
chore: Look for increases in counter for Prometheus alert rule
mvlassis Feb 5, 2026
c748a93
chore: Look for increases in counter for Prometheus alert rule
mvlassis Feb 6, 2026
9388c05
chore: Look for increases in counter for Prometheus alert rule
mvlassis Feb 6, 2026
fd809e2
chore: Look for increases in counter for Prometheus alert rule
mvlassis Feb 6, 2026
478370b
Update Grafana dashboard
mvlassis Feb 11, 2026
e80c249
Update Prometheus description
mvlassis Feb 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions charms/argo-controller/src/grafana_dashboards/basic.json.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -469,7 +469,7 @@
"datasource": "${prometheusds}",
"editorMode": "builder",
"exemplar": false,
"expr": "argo_pod_missing{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}",
"expr": "sum(increase(argo_workflows_pod_missing_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[10m])) by (node_phase)",
"format": "time_series",
"instant": false,
"interval": "",
Expand All @@ -478,7 +478,7 @@
"refId": "A"
}
],
"title": "Workflows missing Pods",
"title": "Workflows missing Pods in the past 10 minutes",
"transparent": true,
"type": "gauge"
},
Expand Down
8 changes: 4 additions & 4 deletions charms/argo-controller/src/prometheus_alert_rules/missing_pods.rule
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
alert: ArgoWorkflowPodsMissing
expr: max_over_time(argo_pod_missing[5m]) > 0
expr: sum without (node_phase, recently_started) (increase(argo_workflows_pod_missing_total[10m])) > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Missing workflow pods detected"
description: >
Detected missing workflow pods in the last 5 minutes.
Missing pods are expected pods that never appeared or were deleted.
See https://argo-workflows.readthedocs.io/en/release-3.7/metrics/#argo_pod_missing for details.
The number of missing workflow pods has increased in the last 5 minutes.
This indicates that expected pods never appeared or were deleted unexpectedly.
See https://argo-workflows.readthedocs.io/en/release-3.7/metrics/#pod_missing for details.
Loading