[gha-runner-scale-set-controller] Prometheus Not Scraping Metrics Endpoint

[jb-2020]

After enabling metrics for the GHA Runner Scale Set Controller[1], I can confirm that the controller and listeners metrics endpoint is properly exposing the metrics. Unfortunately, Prometheus is not scraping the controller or listener(s)[2].

After creating a PodMonitor for these workloads[3], the metrics will begin being scraped. Why is this not included in the helm-chart? Any insight here? I can certainly open up a PR for this.

Is this supposed to work with the out-of-the-box Prometheus configuration?

[1] https://github.com/actions/actions-runner-controller/blob/master/charts/gha-runner-scale-set-controller/values.yaml#L86-L89
[2] https://github.com/actions/actions-runner-controller/blob/master/docs/gha-runner-scale-set-controller/samples/grafana-dashboard/README.md?plain=1#L14
[3]

---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: gha-rs-controller
  namespace: actions-runner-system
spec:
  podMetricsEndpoints:
  - interval: 30s
    targetPort: 8080
    path: /metrics
  namespaceSelector:
    matchNames:
    - actions-runner-system
  selector:
    matchLabels:
      app.kubernetes.io/part-of: gha-rs-controller
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: gha-runner-scale-set
  namespace: actions-runner-system
spec:
  podMetricsEndpoints:
  - interval: 30s
    targetPort: 8080
    path: /metrics
  namespaceSelector:
    matchNames:
    - actions-runner-system
  selector:
    matchLabels:
      app.kubernetes.io/part-of: gha-runner-scale-set

[nikola-jokic]

We decided not to provide out-of-box. The responsibility of the listener and the controller are just to provide a way, so you can collect metrics. Collection can be done in various ways, and different clusters have different ways of collecting metrics. So the decision was that we provide metrics, and the collection of them is completely up to users ☺️