-
Notifications
You must be signed in to change notification settings - Fork 723
Description
What is the bug?
After upgrading from mimir-distributed-5.8.0 to mimir-distributed-6.0.5,
the Ingress for alertmanager paths breaks with:
<error: services "mimir-alertmanager" not found>
This affects all alertmanager ingress paths:
/alertmanager/multitenant_alertmanager/status/multitenant_alertmanager/configs/api/v1/alerts
Root Cause
PR #13064 introduced this breaking change in 6.0.0:
[CHANGE] Ingress: Change default routes to point to non-headless services. #13064
The ingress template was changed to use mimir-alertmanager (ClusterIP) instead
of mimir-alertmanager-headless. However, when alertmanager.zoneAwareReplication.enabled=true,
the chart never creates a mimir-alertmanager ClusterIP service — only zone-specific
services and the headless service are created.
Services that exist with zoneAwareReplication enabled:
mimir-alertmanager-headless ClusterIP None 8080/TCP
mimir-alertmanager-zone-a ClusterIP <IP> 8080/TCP
mimir-alertmanager-zone-b ClusterIP <IP> 8080/TCP
mimir-alertmanager-zone-c ClusterIP <IP> 8080/TCP
Steps to Reproduce
- Install
mimir-distributedchart withalertmanager.zoneAwareReplication.enabled: true - Enable
ingress.enabled: true - Upgrade from any 5.x chart to 6.0.x
- Run
kubectl describe ingress mimir -n <namespace> - Observe
<error: services "mimir-alertmanager" not found>on alertmanager paths
Expected Behavior
Ingress should route alertmanager paths to mimir-alertmanager-headless when
zoneAwareReplication.enabled=true, since the non-headless service is not created
in that configuration.
Actual Behavior
Ingress tries to route to mimir-alertmanager which does not exist, causing all
alertmanager paths to return errors.
Workaround
Override alertmanager ingress paths in values.yaml to explicitly use the headless service:
ingress:
paths:
alertmanager:
alertmanager-headless:
- path: /alertmanager
- path: /multitenant_alertmanager/status
- path: /multitenant_alertmanager/configs
- path: /api/v1/alertsSuggested Fix
The ingress template should conditionally use the headless service for alertmanager
when zoneAwareReplication.enabled=true:
{{- if .Values.alertmanager.zoneAwareReplication.enabled }}
name: mimir-alertmanager-headless
{{- else }}
name: mimir-alertmanager
{{- end }}Environment
| Chart version | mimir-distributed-6.0.5 |
| Mimir version | 3.0.1 |
| Kubernetes | AKS (Azure) |
| Ingress controller | nginx |
| zoneAwareReplication | enabled (3 zones) |
Helm values (relevant section)
alertmanager:
zoneAwareReplication:
enabled: true
zones:
- name: zone-a
...
- name: zone-b
...
- name: zone-c
...
ingress:
enabled: true
hosts:
- mimir.example.comRelevant Links
- PR that introduced the regression: mimir-distributed: don't route ingress to headless services #13064
- CHANGELOG entry: https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/CHANGELOG.md (search
#13064)