How do I setup federation correctly for prometheus ?

[ashutoshraina]

Goal : Federate the prometheus server in istio-system namespace to the prometheus in monitoring namespace.

Steps :

1. Install istio in the default namespace.

$ kubectl get po -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
grafana-b54bb57b9-jz69h                 1/1     Running   0          134m
istio-egressgateway-64bc874f5c-hx8s5    1/1     Running   0          134m
istio-ingressgateway-6b947b8c5d-dlzdn   1/1     Running   0          134m
istio-tracing-9dd6c4f7c-475ml           1/1     Running   0          134m
istiod-654b4b468b-xsqms                 1/1     Running   0          135m
kiali-d45468dc4-499kd                   1/1     Running   0          134m
prometheus-77566c9987-v27bw             2/2     Running   0          134m

The prometheus service, since the ServiceMonitor needs a service.

$ kubectl describe svc prometheus -n istio-system
Name:              prometheus
Namespace:         istio-system
Labels:            app=prometheus
                   install.operator.istio.io/owning-resource=installed-state
                   install.operator.istio.io/owning-resource-namespace=istio-system
                   operator.istio.io/component=AddonComponents
                   operator.istio.io/managed=Reconcile
                   operator.istio.io/version=1.6.7
                   release=istio
Annotations:       prometheus.io/scrape: true
Selector:          app=prometheus
Type:              ClusterIP
IP:                10.96.173.22
Port:              http-prometheus  9090/TCP
TargetPort:        http-prometheus/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

Port forwarding the prometheus service in the istio-system namespace and checking the federate endpoint works as well.

http://localhost:56698/federate?match[]={job!=%22%22}

2. Install kube-prometheus with additional modifications to supply the federation scrape config.

This was done based on hints provided in the istio docs. See here for the details https://istio.io/latest/docs/ops/best-practices/observability/ . Pasting the relevant section below.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-federation
  labels:
    app.kubernetes.io/name: istio-prometheus
spec:
  namespaceSelector:
    matchNames:
    - istio-system
  selector:
    matchLabels:
      app: prometheus
  endpoints:
  - interval: 30s
    scrapeTimeout: 30s
    params:
      'match[]':
      - '{__name__=~"workload:(.*)"}'
      - '{__name__=~"pilot(.*)"}'
    path: /federate
    targetPort: 9090
    honorLabels: true
    metricRelabelings:
    - sourceLabels: ["__name__"]
      regex: 'workload:(.*)'
      targetLabel: "__name__"
      action: replace

local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') +{
    _config+:: {
    namespace: 'monitoring',
    prometheus+:: {
      namespaces+: ['istio-system'],
    },
  },
  prometheus+:: {
    serviceMonitorIstioPrometheus: {
      apiVersion: 'monitoring.coreos.com/v1',
      kind: 'ServiceMonitor',
      metadata: {
        name: 'istio-prometheus',
        namespace: 'monitoring',
      },
      spec: {
        jobLabel: 'job_istio',
        endpoints: [
          {
            honorLabels: true,
            port:'http-prometheus',
            path: '/federate',
            params: {'match[]':['{job!=""}']},
          },
        ],
        selector: {
          matchLabels: {
            app: 'prometheus',
          },
        },
        namespaceSelector:{
          matchNames : ['istio-system'],
        },
      },
    },
  },

};

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

3. This results in the following config ( trimmed down for simplicity ).

- job_name: monitoring/istio-prometheus/0
  honor_labels: true
  honor_timestamps: true
  params:
    match[]:
    - '{job!=""}'
  metrics_path: /federate
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - istio-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: prometheus
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-prometheus
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_job_istio]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-prometheus
    action: replace

Unfortunately, the prometheus service in the default namespace does not show up as a target in the prometheus server in the monitoring namespace.

I tried to hard code the port to 9090 as well which did not work. I tried both with and without the

    _config+:: {
    namespace: 'monitoring',
    prometheus+:: {
      namespaces+: ['istio-system'],
    },

section. It yielded no change.

Can someone help me understand what could be going on here ?