-
Notifications
You must be signed in to change notification settings - Fork 91
Description
Describe the bug
When using Headlamp in a cluster where Grafana Alloy
is installed as a metrics collector and configured to remote_write metrics to an external
Prometheus-compatible endpoint, the plugin displays an error on the Deployment detail page:
"Couldn't detect Prometheus in your cluster."
Alloy does not expose a Prometheus query API (port 9090) within the cluster — it only collects
and forwards metrics. The actual queryable Prometheus endpoint lives outside the cluster.
Expected outcome: either show a more actionable error message suggesting manual configuration,
or support specifying an external HTTP(S) Prometheus URL in the plugin settings.
To Reproduce
Steps to reproduce the bug:
- Install Grafana Alloy in a Kubernetes cluster and configure it to
remote_writeto an
external Prometheus endpoint - Open Headlamp and navigate to any Deployment detail page
- Observe the error banner: "Couldn't detect Prometheus in your cluster."
Environment (please provide info about your environment):
- Installation type: [Mac]
- Headlamp Version: [0.40.1]
- Other: Grafana Alloy installed as metrics collector; metrics are remote_written to an
external Prometheus-compatible backend (no in-cluster Prometheus query API available)
Are you able to fix this issue?
No
Do you want to help fix this issue? Please see our contribution docs. For any non-trivial contributions, we advise you to join our our slack channel to discuss the implementation with Headlamp's core developers.
Additional Context
Since metrics are remote_written to an external Prometheus instance, the in-cluster collector
(Alloy) does not hold long-term data — making the external endpoint the only useful query target.
It would be helpful if the plugin supported specifying a plain HTTP(S) URL
(e.g. https://prometheus.example.com/) in addition to the existing namespace/service:port
format. This is not a hard requirement but a quality-of-life improvement for teams using modern
observability stacks (e.g. Grafana Alloy, OpenTelemetry Collector) where metrics are forwarded
to an external Prometheus-compatible backend.