How to debug issues with receiver_creator ?

[sfc-gh-aivanou]

I am trying to set up receiver_creator, but have issues.
I want to run node_exporter on each of the k8s nodes, and deploy OTEL as Gateway, with receiver_creator, that will generate N receivers, one for each of the nodes. The receivers will then scrape the endpoint provided on each of the nodes.

I use configuration that I attached to run pods. The OTEL collector starts without errors, but does not collect any metrics. Where does the receiver_creator record the errors?

Here is my configuration:

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-agent
  namespace: monitoring
  labels:
    app: opentelemetry
    component: otel-agent
spec:
  selector:
    matchLabels:
      app: opentelemetry
      component: otel-agent
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-agent
    spec:
      serviceAccountName: monitoring-read
      containers:
      - 
        args:
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        - --no-collector.wifi
        - --no-collector.hwmon
        - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
        - --collector.netclass.ignored-devices=^(veth.*)$
        name: node-exporter
        image: prom/node-exporter
        ports:
          - containerPort: 9100
            protocol: TCP
        resources:
          limits:
            cpu: 250m
            memory: 180Mi
          requests:
            cpu: 102m
            memory: 180Mi
        volumeMounts:
        - mountPath: /host/sys
          mountPropagation: HostToContainer
          name: sys
          readOnly: true
        - mountPath: /host/root
          mountPropagation: HostToContainer
          name: root
          readOnly: true
      volumes:
        - hostPath:
            path: /sys
          name: sys
        - hostPath:
            path: /
          name: root
        - configMap:
            name: otel-agent-conf
          name: otel-agent-config-vol
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  namespace: monitoring
  labels:
    app: opentelemetry
    component: otel-collector-conf
data:
  config.yaml: |
    receivers:
      receiver_creator/1:
        watch_observers: [k8s_observer]
        receivers:
          prometheus_simple:
            rule: type == "k8s.node"
            config:
              metrics_path: '/metrics'
              endpoint: '`endpoint`:9100'

    processors:
      batch:
      memory_limiter:
        limit_mib: 1500
        spike_limit_mib: 512
        check_interval: 5s

    extensions:
      k8s_observer:
      host_observer:
      zpages: {}
      memory_ballast:
        size_mib: 683
    exporters:
      prometheus:
        endpoint: "0.0.0.0:9001"
        resource_to_telemetry_conversion:
          enabled: true
      logging:
        loglevel: debug
        sampling_initial: 5
        sampling_thereafter: 200
    service:
      extensions: [zpages, memory_ballast, k8s_observer]
      pipelines:
        metrics/1:
          receivers: [otlp, receiver_creator/1]
          processors: [memory_limiter, batch]
          exporters: [prometheus, logging]
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: monitoring
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: otel-collector
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 1.0.0
spec:
  ports:
  - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
    port: 4317
    protocol: TCP
    targetPort: 4317
  - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
    port: 4318
    protocol: TCP
    targetPort: 4318
  - name: metrics # Default endpoint for querying metrics.
    port: 8888
    targetPort: 8888
  - name: metricsprom # Default endpoint for querying metrics.
    port: 9001
    targetPort: 9001
  selector:
    app.kubernetes.io/name: otel-collector
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: monitoring
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: otel-collector
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 1.0.0
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: exporter
      app.kubernetes.io/name: otel-collector
      app.kubernetes.io/part-of: kube-prometheus
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/component: exporter
        app.kubernetes.io/name: otel-collector
        app.kubernetes.io/part-of: kube-prometheus
        app.kubernetes.io/version: 1.0.0
    spec:
      serviceAccountName: monitoring-read
      securityContext:
        runAsUser: 1000
        runAsGroup: 3000
        fsGroup: 2000
      containers:
      - 
        securityContext:
          privileged: true
        image: otel/opentelemetry-collector-contrib:0.58.0
        name: otel-collector
        resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 200m
            memory: 400Mi
        ports:
        - containerPort: 9001
        volumeMounts:
        - name: otel-collector-config-vol
          mountPath: /etc/otelcol-contrib/config.yaml
          subPath: config.yaml
      volumes:
        - configMap:
            name: otel-collector-conf
          name: otel-collector-config-vol

[sfc-gh-aivanou]

It seems that by default k8s observer does not listen for nodes.
If someone has the same issue, just add the following:

      k8s_observer:
        observe_nodes: true
        observe_pods: true