misconfigured prometheus?

[matharuajay]

So we have configured linkerd to use our own prometheus. And in our prometheus we have configured some custom metrics and added that as additional scrape config in prometheus.

Before linkerd the prometheus used to scrape the endpoint on the specified 8080 port. But after linkerd the prometheus is trying to scrape the metric endpoint at following ports
:4143 -> returns 500
:80 -> returns 502
:4149 -> returns 404
:8080 -> returns 200

Not sure why is it trying to scrape the custom metrics endpoint on those other ports. I do understand that :4143 is linkerd proxy inbound port where it then should redirect the request to target port, but its not doing it?

Also looks like prometheus is trying to scrape the custom metrics on all the open ports.

We are using following versions

• prometheus Prometheus/2.15.2
• linkerd 2.11.3
• kubernetes 1.21

Any insights on how i can prevent it would be super helpful.

Thanks

[adleong]

The docs for configuring Prometheus' scrape config are here https://linkerd.io/2.11/tasks/external-prometheus/