Skip to content

Prometheus plugin shows error when Grafana Alloy is used as metrics collector with remote_write #539

@sho-chan-081

Description

@sho-chan-081

Describe the bug

When using Headlamp in a cluster where Grafana Alloy
is installed as a metrics collector and configured to remote_write metrics to an external
Prometheus-compatible endpoint, the plugin displays an error on the Deployment detail page:

"Couldn't detect Prometheus in your cluster."

Alloy does not expose a Prometheus query API (port 9090) within the cluster — it only collects
and forwards metrics. The actual queryable Prometheus endpoint lives outside the cluster.

Expected outcome: either show a more actionable error message suggesting manual configuration,
or support specifying an external HTTP(S) Prometheus URL in the plugin settings.

To Reproduce

Steps to reproduce the bug:

  1. Install Grafana Alloy in a Kubernetes cluster and configure it to remote_write to an
    external Prometheus endpoint
  2. Open Headlamp and navigate to any Deployment detail page
  3. Observe the error banner: "Couldn't detect Prometheus in your cluster."

Environment (please provide info about your environment):

  • Installation type: [Mac]
  • Headlamp Version: [0.40.1]
  • Other: Grafana Alloy installed as metrics collector; metrics are remote_written to an
    external Prometheus-compatible backend (no in-cluster Prometheus query API available)

Are you able to fix this issue?

No

Do you want to help fix this issue? Please see our contribution docs. For any non-trivial contributions, we advise you to join our our slack channel to discuss the implementation with Headlamp's core developers.

Additional Context

Image

Since metrics are remote_written to an external Prometheus instance, the in-cluster collector
(Alloy) does not hold long-term data — making the external endpoint the only useful query target.

It would be helpful if the plugin supported specifying a plain HTTP(S) URL
(e.g. https://prometheus.example.com/) in addition to the existing namespace/service:port
format. This is not a hard requirement but a quality-of-life improvement for teams using modern
observability stacks (e.g. Grafana Alloy, OpenTelemetry Collector) where metrics are forwarded
to an external Prometheus-compatible backend.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions