You can still query metrics data even if the OpenTelemetry client no longer sends that metrics data

[gaochuang]

Deployment:
opentelemetry client(pod) + opentelmetry collector + prometheus

Function:
Send the CPU usage of multiple processes to the OpenTelemetry Collector through the OpenTelemetry client metrics.

Issue:
When a process stops running, the OpenTelemetry client will not send the CPU usage of that process. However, the OpenTelemetry collector will always query the CPU usage rate of processes, even if they have stopped.

[gaochuang]

@atoulme, Hi atoulm or who can help me about this issue? Thanks in adcance.

[atoulme]

Take a look here: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter
old metrics expire after 5 minutes.

[gaochuang]

On the linux env.
This is my OTEL config.

 receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
      memory_limiter:
        # 80% of maximum memory up to 2G
        limit_mib: 1500
        # 25% of limit up to 2G
        spike_limit_mib: 512
        check_interval: 5s
      k8sattributes:
      k8sattributes/2:
        auth_type: "serviceAccount"
        passthrough: false
        filter:
          node_from_env_var: KUBE_NODE_NAME
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.namespace.name
            - k8s.node.name
            - k8s.pod.start_time
        pod_association:
          - sources:
             from: resource_attribute
             name: k8s.pod.uid
          - sources:
            - from: connection
    extensions:
      zpages: {}
      memory_ballast:
        # Memory Ballast size should be max 1/3 to 1/2 of memory.
        size_mib: 683
    exporters:
      prometheus:
        endpoint: "0.0.0.0:8889"
        enable_open_metrics: true
        resource_to_telemetry_conversion:
          enabled: true
      logging:
      jaeger:
        endpoint: jaeger-all-in-one:14250
      otlp:
        endpoint: "http://someotlp.target.com:4317" # Replace with a real endpoint.
    service:
      extensions: [zpages, memory_ballast]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [jaeger]
        metrics:
          receivers: [otlp]
          processors: [batch, memory_limiter, k8sattributes, k8sattributes/2]
          exporters: [prometheus]

[gaochuang]

Take a look here: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter old metrics expire after 5 minutes.

I use the curl command to get the metrics data from the otel collector's prometheus exporters.
The "example" process has been stopped for more than 12 hours, but its metrics data can still be obtained.

bash-5.1$ curl 172.22.245.240:8889/metrics
# HELP ccs_process_cpu_metrics_v1
# TYPE ccs_process_cpu_metrics_v1 gauge
ccs_process_cpu_metrics_v1{instance="aa1cfde1-5293-4f1d-82aa-92d147c08110",job="xxx_container_metrics",pid="804",process_name="example"} 196.84710693359375 1686450604524
ccs_process_cpu_metrics_v1{instance="aa1cfde1-5293-4f1d-82aa-92d147c08110",job="xxx_container_metrics",pid="1",process_name="xxx"} 0 1686450604406

[gaochuang]

old metrics expire after 5 minutes.

How did you find out the collector is still querying the old processes? Linux or Windows? Got a yaml config file you can share?

Thank you for your quick reply. On the linux Env.

[gaochuang]

804

Take a look here: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter old metrics expire after 5 minutes.

I use the curl command to get the metrics data from the otel collector's prometheus exporters. The "example" process has been stopped for more than 12 hours, but its metrics data can still be obtained.

bash-5.1$ curl 172.22.245.240:8889/metrics # HELP ccs_process_cpu_metrics_v1 # TYPE ccs_process_cpu_metrics_v1 gauge ccs_process_cpu_metrics_v1{instance="aa1cfde1-5293-4f1d-82aa-92d147c08110",job="xxx_container_metrics",pid="804",process_name="example"} 196.84710693359375 1686450604524 ccs_process_cpu_metrics_v1{instance="aa1cfde1-5293-4f1d-82aa-92d147c08110",job="xxx_container_metrics",pid="1",process_name="xxxxx"} 0 1686450604406

After the process is stopped, the obtained metrics data of the process cpu has not been updated.

[gaochuang]

@atoulme Hi, update otel config, as below:
receivers:
logging:
loglevel: debug
otlp:
protocols:
grpc:
http:
processors:
logging:
loglevel: debug
batch:
memory_limiter:
# 80% of maximum memory up to 2G
limit_mib: 1500
# 25% of limit up to 2G
spike_limit_mib: 512
check_interval: 5s
k8sattributes:
k8sattributes/2:
auth_type: "serviceAccount"
passthrough: false
filter:
node_from_env_var: KUBE_NODE_NAME
extract:
metadata:
- k8s.pod.name
- k8s.pod.uid
- k8s.deployment.name
- k8s.namespace.name
- k8s.node.name
- k8s.pod.start_time
pod_association:
- sources:
from: resource_attribute
name: k8s.pod.uid
- sources:
- from: connection
extensions:
zpages: {}
memory_ballast:
# Memory Ballast size should be max 1/3 to 1/2 of memory.
size_mib: 683
exporters:
logging:
loglevel: debug
prometheus:
endpoint: "0.0.0.0:8889"
enable_open_metrics: true
metric_expiration: 5m
send_timestamps: true
resource_to_telemetry_conversion:
enabled: true
jaeger:
endpoint: 172.22.121.119:14250
otlp:
endpoint: "http://someotlp.target.com:4317" # Replace with a real endpoint.
service:
extensions: [zpages, memory_ballast]
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [jaeger]
metrics:
receivers: [otlp]
processors: [batch, memory_limiter, k8sattributes, k8sattributes/2]
exporters: [logging, prometheus]

Issue still exists. And the timestamp keeps updating

[gaochuang]

@open-telemetry/collector-contrib-maintainer Hi, OTEL experts, Can anyone help me to look at this issue? Thanks.