How to add node ip address as label to metrics?

[sfc-gh-aivanou]

Hello! I am trying to set up the otel on my k8s cluster to capture host metrics and have a question:

I am setting up otel in a gateway mode and scraping hostmetrics. How do I add node ip address as label to each of the separate metric streams?

The configuration below produces the following metric output:

system_cpu_time{container="otel-collector", cpu="cpu0", endpoint="metricsprom", instance="10.244.20.60:9001", job="otel-collector", namespace="monitoring", pod="otel-collector-85cf4f94d9-fnkbx", service="otel-collector", state="idle"}

Where instance is the address of the gateway collector, which is not what I am looking for.

Here is my config:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: monitoring-read
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: 
    - ""
    - "batch"
    - "apps"
    - "autoscaling"
  resources:
  - nodes
  - nodes/metrics
  - nodes/proxy
  - services
  - endpoints
  - pods
  - jobs
  - batch
  - resourcequotas
  - deployments
  - statefulsets
  - daemonsets
  - cronjobs
  - replicasets
  - namespaces
  - replicationcontrollers
  - horizontalpodautoscalers
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: monitoring-read
  namespace: monitoring
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-agent-conf
  namespace: monitoring
  labels:
    app: opentelemetry
    component: otel-agent-conf
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      hostmetrics:
        collection_interval: 30s
        scrapers:
          cpu:
          load:
          memory:
          disk:
          filesystem:
          network:

    exporters:
      logging:
        loglevel: debug
        sampling_initial: 5
        sampling_thereafter: 200

      otlp:
        endpoint: "otel-collector.monitoring:4317"
        tls:
          insecure: true
        sending_queue:
          num_consumers: 4
          queue_size: 100
        retry_on_failure:
          enabled: true

    processors:
      batch:
      memory_limiter:
        # 80% of maximum memory up to 2G
        limit_mib: 400
        # 25% of limit up to 2G
        spike_limit_mib: 100
        check_interval: 5s
    extensions:
      zpages: {}
      memory_ballast:
        # Memory Ballast size should be max 1/3 to 1/2 of memory.
        size_mib: 165
    service:
      telemetry:
        logs:
          level: "info"
      extensions: [zpages, memory_ballast]
      pipelines:
        metrics:
          receivers: [hostmetrics]
          processors: [batch, memory_limiter]
          exporters: [otlp, logging]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-agent
  namespace: monitoring
  labels:
    app: opentelemetry
    component: otel-agent
spec:
  selector:
    matchLabels:
      app: opentelemetry
      component: otel-agent
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-agent
    spec:
      serviceAccountName: monitoring-read
      containers:
      - 
        env:
        - name: KUBE_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: otel/opentelemetry-collector-contrib:0.54.0
        name: otel-agent
        resources:
          limits:
            cpu: 500m
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 55679 # ZPages endpoint.
        - containerPort: 4317 # Default OpenTelemetry receiver port.
        - containerPort: 8888  # Metrics.
        volumeMounts:
        - name: otel-agent-config-vol
          mountPath: /etc/otelcol-contrib/config.yaml
          subPath: config.yaml
      volumes:
        - configMap:
            name: otel-agent-conf
          name: otel-agent-config-vol
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  namespace: monitoring
  labels:
    app: opentelemetry
    component: otel-collector-conf
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:      
    processors:
      batch:
      memory_limiter:
        limit_mib: 1500
        spike_limit_mib: 512
        check_interval: 5s

    extensions:
      zpages: {}
      memory_ballast:
        # Memory Ballast size should be max 1/3 to 1/2 of memory.
        size_mib: 683
    exporters:
      prometheus:
        endpoint: "0.0.0.0:9001"
        resource_to_telemetry_conversion:
          enabled: true
      logging:
        loglevel: debug
        sampling_initial: 5
        sampling_thereafter: 200
    service:
      extensions: [zpages, memory_ballast]
      pipelines:
        metrics/1:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [prometheus, logging]
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: monitoring
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: otel-collector
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 1.0.0
spec:
  ports:
  - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
    port: 4317
    protocol: TCP
    targetPort: 4317
  - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
    port: 4318
    protocol: TCP
    targetPort: 4318
  - name: metrics # Default endpoint for querying metrics.
    port: 8888
    targetPort: 8888
  - name: metricsprom # Default endpoint for querying metrics.
    port: 9001
    targetPort: 9001
  selector:
    app.kubernetes.io/name: otel-collector
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: monitoring
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: otel-collector
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 1.0.0
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: exporter
      app.kubernetes.io/name: otel-collector
      app.kubernetes.io/part-of: kube-prometheus
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  replicas: 1 
  template:
    metadata:
      labels:
        app.kubernetes.io/component: exporter
        app.kubernetes.io/name: otel-collector
        app.kubernetes.io/part-of: kube-prometheus
        app.kubernetes.io/version: 1.0.0
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 3000
        fsGroup: 2000
      containers:
      - 
        securityContext:
          privileged: true
        image: otel/opentelemetry-collector-contrib:0.54.0
        name: otel-collector
        resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 200m
            memory: 400Mi
        ports:
        - containerPort: 9001
        volumeMounts:
        - name: otel-collector-config-vol
          mountPath: /etc/otelcol-contrib/config.yaml
          subPath: config.yaml
      volumes:
        - configMap:
            name: otel-collector-conf
          name: otel-collector-config-vol

[jtama]

You could use K8s downward api and ressource processor with env detector ?

Add the following env in your daemonset

- name: NODE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
- name : OTEL_RESOURCE_ATTRIBUTES
  value: "instance=${NODE_IP}"

And in your otel conf :

processors:
      resourcedetection:
        detectors: [env]