COS integration passes the wrong target IP

[barryprice]

Bug Description

Hi,

We've got this deployed and related/integrated with COS, but are seeing an error due to COS's inability to scrape the defined Prometheus target.

On closer investigation, the source of the error is an inability to scrape the actual pod IP, rather than the service/ingress IP.

The configured target is http://192.168.102.222:9102/metrics

I would expect the charm to pass through the ingress IP (or possibly the service ClusterIP?) as the defined target instead - these are in the 10.0.0.0/8 range rather than 192.168.0.0/16.

To Reproduce

1. Deploy and expose httprequest-lego-provider via nginx-ingress-integrator.

2. Integrate the service with COS.

3. Failure to scrape metrics target.

Environment

Deployed on a shared Kubernetes cluster on Openstack.

Relevant log output

Nothing obviously relevant on this side, but feel free to request specific logs.

Additional context

No response

[javierdelapuente]

@arturo-seijas can you have a look at this? Thanks

[arturo-seijas]

@barryprice , I would expect juju to deal with networking here. Where is the Prometheus instance deployed? Is it outside the cluster?

[barryprice]

Both COS itself (including Prometheus) and the problem environment are deployed on the same cloud, but in separate environments, on separate k8s clusters, if that's what you mean?

[arturo-seijas]

@javierdelapuente , this is a PAAS application. Is this supported? Should I file a ticket there?

[javierdelapuente]

@barryprice , if I understand correctly, then the proper way to do this is to use a grafana-agent https://charmhub.io/grafana-agent-k8s.

It is correct that the IPs to scrape are the pods IPs (scraping the ingress could get random Pods).

[barryprice]

Perhaps this would be better targeted as a documentation issue then?

Relating the application with COS ought to provide useful monitoring data, but in this case it just provides unactionable alerts.

Should the documentation call out that COS relations should only be attempted if COS is deployed on the same Kubernetes cluster as the application (with network policies set to allow traffic between both namespaces to the target pods directly)?

[javierdelapuente]

I am not really sure where this should go.

This will happen with any deployment that cannot be scraped directly... we are also using the grafana-agent also in machine deployments.

But as you say, it is not clear what to do...

[arturo-seijas]

Perhaps this should be documented in the prometheus charm?

[barryprice]

Our observability folks are probably better qualified to chime in on this, as it's an issue that I suspect won't necessarily be specific to this charm? @sajoupa @mcarvalhor @alexdlukens

[mcarvalhor]

I believe that there should be a mechanism in-place to prevent or warn about this, maybe the charm should enter a blocked - you need to use grafana-agent state/message?

[arturo-seijas]

@mcarvalhor I don't think the charm has any way to know its metrics endpoint will be scraped by a Prometheus instance outside of the cluster

[lucabello]

Luca from the Observability team, leaving my two cents here :)

We've had the debate between scraping things over the internal (usually k8s fqdn) or external (ingressed) URL for our own charms' self-monitoring, and we ended up going with the external one for reasons (mainly TLS-related).

The workflow we recommend is to indeed use grafana-agent to scrape things outside the COS model, and then relate that to COS. With that deployment, a user won't have to worry about which URL this charm is using. We have some docs about it, and this is IMHO a documentation problem, but I'm not sure what the best way to go about it is.

Maybe @sed-i can provide better insight :)

[sed-i]

@lucabello is correct: scrape is for in-model, remote-write is for cross-model.
We reached that design following the discussion here.
Recently a new juju feature - remote_model_get - was introduced, and we plan to start using it when we move our pin from juju >=3.1 to 3.6.

[mcarvalhor]

Thanks a lot! I believe this issue for httprequest-lego-provider can be closed.