inconsistent label cardinality: expected 6 label values but got 7 in prometheus.Labels

[whitehatbuds]

2025/09/09 06:28:27 http: panic serving 10.65.12.38:63456: inconsistent label cardinality: expected 6 label values but got 7 in prometheus.Labels{"kind":"tag", "project":"data/dsf/recruitment/recruitment-admin-grpc", "ref":"chart@1.0.3", "source":"push", "status":"success", "topics":"", "variables":""}
goroutine 48343 [running]:
net/http.(*conn).serve.func1()
net/http/server.go:1947 +0xbe
panic({0xfd7c80?, 0xc04dfb32a0?})
runtime/panic.go:785 +0x132
github.com/prometheus/client_golang/prometheus.(*GaugeVec).With(...)
github.com/prometheus/client_golang@v1.20.5/prometheus/gauge.go:250
github.com/mvisonneau/gitlab-ci-pipelines-exporter/pkg/controller.(*Registry).ExportMetrics(0xc03ba08a00, 0x132a2f0?)
github.com/mvisonneau/gitlab-ci-pipelines-exporter/pkg/controller/metrics.go:194 +0x206
github.com/mvisonneau/gitlab-ci-pipelines-exporter/pkg/controller.(*Controller).MetricsHandler(0xc00024c608, {0x1328908, 0xc0005e2000}, 0xc00baea000)
github.com/mvisonneau/gitlab-ci-pipelines-exporter/pkg/controller/handlers.go:57 +0x42c
net/http.HandlerFunc.ServeHTTP(0xc0001ea380?, {0x1328908?, 0xc0005e2000?}, 0x8ea396?)
net/http/server.go:2220 +0x29
net/http.(*ServeMux).ServeHTTP(0x46e479?, {0x1328908, 0xc0005e2000}, 0xc00baea000)
net/http/server.go:2747 +0x1ca
net/http.serverHandler.ServeHTTP({0xc00fcb2390?}, {0x1328908?, 0xc0005e2000?}, 0x6?)
net/http/server.go:3210 +0x8e
net/http.(*conn).serve(0xc00681e120, {0x132a2b8, 0xc000c18e70})
net/http/server.go:2092 +0x5d0
created by net/http.(*Server).Serve in goroutine 431
net/http/server.go:3360 +0x485

Image version: v0.5.10
how do I identify which config i need to change to fix this error?

[whitehatbuds]

How do i tell which GaugeMetric name is the culprit for this.

Does this panic also result in 503 when i scrape /metrics endpoint

[whitehatbuds]

server:
  webhook:
    enabled: true

gitlab:
  url: xxx
  health_url: xxx/-/health
  maximum_requests_per_second: 10
  burstable_requests_per_second: 20
  maximum_jobs_queue_size: 40000

wildcards:
- owner:
    include_subgroups: true
    kind: group
    name: risk
  pull:
    refs:
      branches:
        regexp: "^release.*"
    pipeline:
      variables:
        enabled: true
- owner:
    include_subgroups: true
    kind: group
    name: data/dsf
  pull:
    refs:
      branches:
        regexp: "(main|master)"
- owner:
    include_subgroups: true
    kind: group
    name: md
  pull:
    refs:
      branches:
        regexp: ".*"
      merge_requests:

    pipeline:
      test_reports:
        enabled: true

# Enable metrics to be pulled and accepted by the webhook
project_defaults:
  pull:
    environments:
      enabled: true
    refs:
      branches:
        enabled: true
      tags:
        enabled: true
      merge_requests:
        enabled: false
    pipeline:
      jobs:
        enabled: false
      variables:
        enabled: false
      test_reports:
        enabled: false

# Pull-based approach
pull:
  projects_from_wildcards:
    on_init: true
    scheduled: true
    interval_seconds: 86400 # Default 1800

  environments_from_projects:
    on_init: true
    scheduled: true
    interval_seconds: 43200 # Default 300

  refs_from_projects:
    on_init: true
    scheduled: true
    interval_seconds: 43200 # Default 300

  metrics:
    on_init: true
    scheduled: true
    interval_seconds: 43200 # Default 30

# Garbage collection for pull-based approach
garbage_collect:
  projects:
    on_init: true
    scheduled: true
    interval_seconds: 172800

  environments:
    on_init: true
    scheduled: true
    interval_seconds: 86400

  refs:
    on_init: true
    scheduled: true
    interval_seconds: 86400

  metrics:
    on_init: true
    scheduled: true
    interval_seconds: 86400

These are my configs

[whitehatbuds]

Problem seems to go away after restarting the redis master pod. What could have caused this?
My redis config uses

redis:
  persistent: false

I looked through past issues, the other user was facing something related to the "username" label.

[whitehatbuds]

If exporter was upgraded, and there were changes to the metric schema,

NewCollector runs on startup to update the expected metric labels. If it already exists in redis, and redis is not wiped

func (r *Redis) Metrics(ctx context.Context) (schemas.Metrics, error) {
metrics := schemas.Metrics{}

marshalledMetrics, err := r.HGetAll(ctx, redisMetricsKey).Result()
if err != nil {
	return metrics, err
}

for stringMetricKey, marshalledMetric := range marshalledMetrics {
	m := schemas.Metric{}

	if err := msgpack.Unmarshal([]byte(marshalledMetric), &m); err != nil {
		return metrics, err
	}

	metrics[schemas.MetricKey(stringMetricKey)] = m
}

return metrics, nil

}

The cached labels are used instead and its what causes the error?

[whitehatbuds]

i think the panic happened again, anyone know why it comes back after redis has been restarted?

[StickMan077]

I'm experiencing the same issue — it happens daily in my case