How to use kube-prometheus to monitoring an external kubernetes?

[DesireWithin]

Hello,

Scenario is that I have a kube-prometheus in k8s-A. And then I want to monitoring another k8s-B. Both k8s-A and k8s-B are not a big k8s cluster.
Here is my questions:
Accorinding to this blog, add extenal service by using service+endpoints+serviceMonitor only support IP, so I can't expose service of k8s-B by ingress. Which means I have to use a lot of NodePort Service. If i want to monitoring coredns of k8s-B, I have to separate metrics port from existing coredns service, because a simple coredns service is like this:

...
  clusterIP: 10.96.0.10
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  - name: metrics
    port: 9153
    protocol: TCP
    targetPort: 9153
...

So what is the best way to use kube-prometheus to monitoring multiple k8s?
Is Prometheus Federation an optional choose?

• Prometheus-master
  • kube-prometheus-A in K8S-A
  • kube-prometheus-B in K8S-B

Let Prometheus-master pull metrics from kube-prometheus-A and kube-prometheus-B separately, is it a good solution?

[bryanasdev000]

From experience, I would leave the Prometheus separate in each cluster but writing via remote_write to a single point and from there it would be trivial for Grafana to read, this would need projects like Cortex, Thanos, VictoriaMetrics or similar.

The Federation of Prometheus can also be an option, you will have a Prometheus that will be the gateway to any other, so you only need to expose the Prometheus of each cluster to this hub and Grafana would read from it only.

Depending on how you do to monitor externally, you give up a part of the operator automation to do things manually (or manage other automation), where basically you will need to expose everything you want to monitor so that Cluster A can arrive at the required Cluster B endpoints, plus RBAC and network adjustments.

For more information, you can start with these links:

• Monitoring kubernetes with prometheus from outside of k8s cluster. prometheus/prometheus#4633

• https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

• https://prometheus.io/docs/prometheus/latest/federation/

• https://prometheus.io/docs/operating/integrations/