Skip to content

[helm] Ingress routes to non-existent mimir-alertmanager service when zoneAwareReplication is enabled #14334

@dehimb

Description

@dehimb

What is the bug?

After upgrading from mimir-distributed-5.8.0 to mimir-distributed-6.0.5,
the Ingress for alertmanager paths breaks with:

<error: services "mimir-alertmanager" not found>

This affects all alertmanager ingress paths:

  • /alertmanager
  • /multitenant_alertmanager/status
  • /multitenant_alertmanager/configs
  • /api/v1/alerts

Root Cause

PR #13064 introduced this breaking change in 6.0.0:

[CHANGE] Ingress: Change default routes to point to non-headless services. #13064

The ingress template was changed to use mimir-alertmanager (ClusterIP) instead
of mimir-alertmanager-headless. However, when alertmanager.zoneAwareReplication.enabled=true,
the chart never creates a mimir-alertmanager ClusterIP service — only zone-specific
services and the headless service are created.

Services that exist with zoneAwareReplication enabled:

mimir-alertmanager-headless   ClusterIP   None    8080/TCP 
mimir-alertmanager-zone-a     ClusterIP   <IP>   8080/TCP 
mimir-alertmanager-zone-b     ClusterIP   <IP>    8080/TCP 
mimir-alertmanager-zone-c     ClusterIP   <IP>   8080/TCP 

Steps to Reproduce

  1. Install mimir-distributed chart with alertmanager.zoneAwareReplication.enabled: true
  2. Enable ingress.enabled: true
  3. Upgrade from any 5.x chart to 6.0.x
  4. Run kubectl describe ingress mimir -n <namespace>
  5. Observe <error: services "mimir-alertmanager" not found> on alertmanager paths

Expected Behavior

Ingress should route alertmanager paths to mimir-alertmanager-headless when
zoneAwareReplication.enabled=true, since the non-headless service is not created
in that configuration.

Actual Behavior

Ingress tries to route to mimir-alertmanager which does not exist, causing all
alertmanager paths to return errors.

Workaround

Override alertmanager ingress paths in values.yaml to explicitly use the headless service:

ingress:
  paths:
    alertmanager:
    alertmanager-headless:
      - path: /alertmanager
      - path: /multitenant_alertmanager/status
      - path: /multitenant_alertmanager/configs
      - path: /api/v1/alerts

Suggested Fix

The ingress template should conditionally use the headless service for alertmanager
when zoneAwareReplication.enabled=true:

{{- if .Values.alertmanager.zoneAwareReplication.enabled }}
name: mimir-alertmanager-headless
{{- else }}
name: mimir-alertmanager
{{- end }}

Environment

Chart version mimir-distributed-6.0.5
Mimir version 3.0.1
Kubernetes AKS (Azure)
Ingress controller nginx
zoneAwareReplication enabled (3 zones)

Helm values (relevant section)

alertmanager:
  zoneAwareReplication:
    enabled: true
    zones:
      - name: zone-a
        ...
      - name: zone-b
        ...
      - name: zone-c
        ...

ingress:
  enabled: true
  hosts:
    - mimir.example.com

Relevant Links

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelmhelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions