Dropping instance label with attributesprocessor

[imranismail]

I can't seem to be able to drop the instance label from all my data collected. I can't get this to work with the attributesprocessor plugin.

Is this attribute reserved somehow?

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: opentelemetry

resources:
  - namespace.yaml

components:
  - ../../image-pull-secrets/base/components/secrets

helmCharts:
  - &chartDefault # opentelemetry-collector-standalone
    releaseName: &releaseName opentelemetry-collector-standalone
    name: opentelemetry-collector
    namespace: opentelemetry
    repo: https://open-telemetry.github.io/opentelemetry-helm-charts
    version: 0.53.0
    valuesInline:
      mode: deployment
      priorityClassName: system-cluster-critical
      imagePullSecrets: &imagePullSecrets
        - name: registry-docker
      autoscaling:
        enabled: true
        minReplicas: 3
        maxReplicas: 20
        targetCPUUtilizationPercentage: 80
        targetMemoryUtilizationPercentage: 80
      presets:
        kubernetesAttributes:
          enabled: true
        clusterMetrics:
          enabled: false
      resources:
        requests:
          cpu: 256m
          memory: 2.5Gi
        limits:
          cpu: 512m
          memory: 2.5Gi
      config:
        receivers:
          k8s_cluster:
            collection_interval: &collection_interval 30s
          prometheus:
            config:
              global:
                scrape_interval: *collection_interval
              scrape_configs:
                ##
                # Scrape from self
                ##
                - job_name: *releaseName
                  static_configs:
                    - targets:
                        - ${MY_POD_IP}:8888
        processors:
          filter/kube-probe:
            spans:
              exclude:
                match_type: regexp
                attributes:
                  - key: http.user_agent
                    value: kube-probe\/.+
          attributes/common:
            actions:
              - key: organization
                action: insert
                value: "${ORGANIZATION}"
              - key: environment
                action: insert
                value: "${ENVIRONMENT}"
              - key: cloud
                action: insert
                value: "${CLOUD}"
              - key: region
                action: insert
                value: "${REGION}"
             # THIS DOESN'T WORK
              - key: instance
                action: delete
          attributes/cadvisor:
            include:
              match_type: regexp
              metric_names:
                - ^container.*
            actions:
              - key: id
                action: delete
              - key: image
                action: delete
              - key: name
                action: delete
          attributes/kube-state-metrics:
            include:
              match_type: regexp
              metric_names:
                - ^kube.*
            actions:
              - key: uid
                action: delete
              - key: container_id
                action: delete
              - key: image_id
                action: delete
          batch:
            send_batch_size: 4096
            send_batch_max_size: 4096
        exporters:
          prometheusremotewrite:
            endpoint: ${PROMETHEUS_REMOTE_WRITE_ENDPOINT}
            headers:
              authorization: Bearer ${PROMETHEUS_REMOTE_WRITE_TOKEN}
          otlp:
            endpoint: ${OTLP_ENDPOINT}
            headers:
              authorization: "Bearer ${OTLP_TOKEN}"
            compression: snappy
        service:
          pipelines:
            metrics:
              receivers:
                - otlp
                - prometheus
              processors:
                - k8sattributes
                - attributes/common
                - attributes/cadvisor
                - attributes/kube-state-metrics
                - memory_limiter
                - batch
              exporters:
                - prometheusremotewrite
            traces:
              receivers:
                - otlp
              processors:
                - filter/kube-probe
                - k8sattributes
                - attributes/common
                - memory_limiter
                - batch
              exporters:
                - otlp

  - <<: *chartDefault # opentelemetry-collector
    releaseName: &releaseName opentelemetry-collector
    valuesInline:
      mode: daemonset
      tolerations:
        - operator: Exists
      priorityClassName: system-node-critical
      imagePullSecrets: *imagePullSecrets
      presets:
        kubernetesAttributes:
          enabled: true
        hostMetrics:
          enabled: true
      extraEnvs:
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
      resources:
        requests:
          cpu: 128m
          memory: 2Gi
        limits:
          cpu: 256m
          memory: 2Gi
      config:
        receivers:
          hostmetrics:
            collection_interval: *collection_interval
          prometheus:
            config:
              global:
                scrape_interval: *collection_interval
              scrape_configs:
                ##
                # Scrape metrics from self
                ##
                - job_name: *releaseName
                  static_configs:
                    - targets:
                        - ${MY_POD_IP}:8888

                ##
                # Scrape metrics from `/metrics/cadvisor` endpoint
                ##
                - job_name: kubernetes-cadvisor
                  scheme: https
                  tls_config:
                    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  authorization:
                    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                  kubernetes_sd_configs:
                    - role: node
                  metrics_path: /metrics/cadvisor
                  relabel_configs:
                    - action: keep
                      regex: "${K8S_NODE_NAME}"
                      source_labels:
                        - __meta_kubernetes_node_name

                ##
                # Scrape metrics from all pods annotated with prometheus.io/scrape=true
                ##
                - job_name: kubernetes-pods
                  kubernetes_sd_configs:
                    - role: pod
                  relabel_configs:
                    - action: keep
                      regex: "true;${K8S_NODE_NAME}"
                      source_labels:
                        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
                        - __meta_kubernetes_pod_node_name
                    - action: replace
                      regex: (https?)
                      source_labels:
                        - __meta_kubernetes_pod_annotation_prometheus_io_scheme
                      target_label: __scheme__
                    - action: replace
                      regex: (.+)
                      source_labels:
                        - __meta_kubernetes_pod_annotation_prometheus_io_path
                      target_label: __metrics_path__
                    - action: replace
                      regex: ([^:]+)(?::\d+)?;(\d+)
                      replacement: $$1:$$2
                      source_labels:
                        - __address__
                        - __meta_kubernetes_pod_annotation_prometheus_io_port
                      target_label: __address__
                    - source_labels:
                        - __meta_kubernetes_pod_controller_kind
                        - __meta_kubernetes_namespace
                        - __meta_kubernetes_pod_controller_name
                      separator: /
                      target_label: job
                    - source_labels:
                        - job
                      regex: ^ReplicaSet/([a-z0-9-]+)/([a-z0-9-]+)-[a-z0-9]+$
                      replacement: Deployment/$$1/$$2
                      target_label: job

                ##
                # Scrape metrics from all services annotated with prometheus.io/scrape=true
                ##
                - job_name: kubernetes-endpoints
                  kubernetes_sd_configs:
                    - role: endpoints
                  relabel_configs:
                    - action: keep
                      regex: "true;${K8S_NODE_NAME}"
                      source_labels:
                        - __meta_kubernetes_service_annotation_prometheus_io_scrape
                        - __meta_kubernetes_endpoint_node_name
                    - action: replace
                      regex: (.+)
                      source_labels:
                        - __meta_kubernetes_service_annotation_prometheus_io_path
                      target_label: __metrics_path__
                    - action: replace
                      regex: ([^:]+)(?::\d+)?;(\d+)
                      replacement: $$1:$$2
                      source_labels:
                        - __address__
                        - __meta_kubernetes_service_annotation_prometheus_io_port
                      target_label: __address__
                    - source_labels:
                        - __meta_kubernetes_namespace
                        - __meta_kubernetes_service_name
                      separator: /
                      regex: ^([a-z0-9-]+)/([a-z0-9-]+)$
                      replacement: Service/$$1/$$2
                      target_label: job
        processors:
          k8sattributes:
            filter:
              node_from_env_var: K8S_NODE_NAME
            passthrough: true
          batch:
            send_batch_size: 4096
            send_batch_max_size: 4096
        exporters:
          otlp:
            endpoint: opentelemetry-collector-standalone:4317
            tls:
              insecure: true
            compression: snappy
        service:
          pipelines:
            metrics:
              processors:
                - k8sattributes
                - memory_limiter
                - batch
              exporters:
                - otlp
            traces:
              processors:
                - k8sattributes
                - memory_limiter
                - batch
              exporters:
                - otlp
      clusterRole:
        rules:
          - apiGroups: [""]
            resources:
              - nodes
              - nodes/metrics
              - endpoints
              - services
            verbs: ["get", "list", "watch"]
          - nonResourceURLs: ["/metrics"]
            verbs: ["get", "watch", "list"]

[mar4uk]

job and instance labels are set by the prometheusremotewrite exporter. The exporter is the last one in the collector pipeline, that's why processor can't remove these attributes.

Here is some details on why job and instance labels should be sent to Prometheus

Hope it helps, let me know if it's not