Discover all /metrics endpoints to be scraped

[ajcasagrande]

This would allow the user to just run AIPerf and all urls that need to be scraped will be auto-detected

Things to consider:

• Kuberentes vs SLURM vs Local, what each would support for us to do this
• Assume Prometheus collection service is not running

[ardecode]

@ajcasagrande , I can take this.

[ardecode]

@ajcasagrande I am planning to break this into 3 PRs

• PR1 (Local) probes common paths (/metrics, /prometheus/metrics, /v1/metrics, /api/metrics, /api/v1/metrics) on the provided url hosts.
• PR2 (Kubernetes) queries the K8s API directly for pods with prometheus.io/* annotations to discover hosts and paths (no dependency on Prometheus)
• PR3 (SLURM) parses SLURM_JOB_NODELIST to get allocated nodes, then probes common ports and paths on each.

Does this breakdown align with what you had in mind?

[ajcasagrande]

@ardecode I think the k8s api one is best to tackle first. You can also look at I think it was http_sd_config.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config

Local I'm not really sure what we can do there that much, unless the endpoint provides us with something to use to discover.

This feature is more about discovering the individual frontends, prefill workers, decode workers, agg workers, etc. in a distributed environment such that you dont need to list every single one to --server-metrics

So i would say priority 1 would be k8s, then slurm, then local.

[ardecode]

Thanks for the pointer to http_sd_config.

For k8s auto-discovery where AIPerf scraping metrics and assuming no Prometheus running, what explicit opt-in signal should AIPerf use to decide which pods expose /metrics and should be scraped? A label selector, Prometheus-style annotations, or something else?

If I am not wrong Dynamo deployments provide signals like nvidia.com/metrics-enabled=true and component labels, but for non-Dynamo setups like vLLM I dont think there is a standard convention today. Should AIPerf rely on user-provided selectors/annotations in those cases, or define a generic AIPerf-level convention?

[ajcasagrande]

@ardecode I'm not the most familiar in this landscape, but this is what claude came up with, and I think its fairly good starting reference:

TL;DR

Use a layered approach that checks multiple conventions in priority order:

1. Dynamo labels (for NVIDIA Dynamo users)
2. Prometheus annotations (industry standard, broadest compatibility)
3. User-provided selector (fallback for custom setups)

Do not define an AIPerf-specific convention - leverage existing standards instead.

---

Recommended Discovery Hierarchy

Priority	Signal	Use Case
1	nvidia.com/metrics-enabled: "true" (label)	Dynamo deployments - automatically added by operator
2	prometheus.io/scrape: "true" (annotation)	Non-Dynamo setups (vLLM, SGLang, etc.) - widely adopted standard
3	--metrics-label-selector="app=vllm" (CLI flag)	Custom deployments with no standard signals

---

Why This Order?

Priority 1: Dynamo Labels

• Dynamo operator automatically adds nvidia.com/metrics-enabled: "true" to all pods
• Also provides nvidia.com/dynamo-component-type: "frontend|worker" for component identification
• Zero configuration required for Dynamo users

Priority 2: Prometheus Annotations

• De facto industry standard - used by Prometheus, Grafana, Datadog, and most observability tools
• Already familiar to platform teams
• Includes additional metadata:

Annotation	Purpose	Default
prometheus.io/scrape	Opt-in signal	"true"
prometheus.io/port	Metrics port	Container port
prometheus.io/path	Metrics path	/metrics
prometheus.io/scheme	Protocol	http

Priority 3: User-Provided Selector

• Fallback for deployments without standard signals
• Allows users to target any label combination
• Example: --metrics-label-selector="app=my-inference-server,env=prod"