Skip to content

Labels only being scraped for some resources #2791

@cedws

Description

@cedws

What happened: We have a common label across many of our K8s resources that looks like company.io/support-tier. Via the kube-prometheus-stack chart we have configured kube-state-metrics to scape this label with --metric-labels-allowlist=*=[company.io/support-tier], and have tried other variants of this like just --metric-labels-allowlist=deployments=[company.io/support-tier]. However this label only seems to be getting scraped for a subset of resource types/resources. On our dev cluster we have 10+ deployments with this label but querying kube_deployment_labels shows only two of them with it.

What you expected to happen: Labels to be scraped for all deployments/all resources.

How to reproduce it (as minimally and precisely as possible):

I'm not able to reproduce it on a local cluster which complicates debugging this issue.

I've been checking kube-state-metrics /metrics endpoint directly to eliminate Prometheus itself as the issue.

  1. Create deployment with company.io/support-tier label
  2. Configure kube-state-metrics with --metric-labels-allowlist=deployments=[company.io/support-tier]
  3. Look at /metrics and see if the deployment has the label

Anything else we need to know?:

Environment:

  • kube-state-metrics version: v2.17.0
  • Kubernetes version (use kubectl version): v1.33.4-eks-e386d34
  • Cloud provider or hardware configuration: EKS
  • Other info:

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    Status

    Needs Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions