Getting API Server metrics for AWS EKS using Prometheus receiver

[jvijayan6]

Hi Team,

This time I was trying to get the metrics for kubernetes-apiserver from AWS EKS. My first attempt was to use this smartagent/kubernetes-apiserver: but EKS not supported this configuration.
After the R&D found Prometheus can be used to get the metrices from KUBERNETES_SERVICE_HOST. But this also not giving any output. Below is the current configuration used in otel-k8s-cluster-receiver, need an extra pair of eyes on this

    receivers:
      prometheus/k8s_apiserver_metrics:
        config:
          scrape_configs:
          - job_name: 'kubernetes-apiservers'
            scheme: https
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            authorization:
              credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true
            scrape_interval: 10s
            static_configs:
            - targets:
              - ${KUBERNETES_SERVICE_HOST}:443
    service:
      pipelines:
        metrics:
          receivers:
            - k8s_cluster
            - prometheus/k8s_apiserver_metrics

@dmitryax @jvoravong

[jvijayan6]

@dmitryax @jvoravong

[dmitryax]

@jvijayan6

LGTM, but you need to remove the following section. That label isn't set since you scrape from a static endpoint, not a discovered pod

            relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true

Also, keep in mind that it'll scrape all the metrics exposed by the k8s API while smartagent/kubernetes-apiserver scrapes only the metrics marked as defaults. If you want to reduce the metrics set, you'd need to apply some metric_relabel_configs to drop what you don't need.

BTW this Q should be raised in https://github.com/signalfx/splunk-otel-collector-chart, not in this repo

[jvijayan6]

thanks a lot @dmitryax