high lock contention wait

[zdyj3170101136]

what i do?

use datadog to profiling logstash.

logstash version

8.15.0

logstash config

logstashConfig:
  logstash.yml: |
    http.host: "0.0.0.0"
    queue.type: persisted
    queue.drain: true
    queue.max_bytes: 1024mb
    config.reload.automatic: true
    config.reload.interval: 30s
    pipeline.batch.size: 4096
    pipeline.ecs_compatibility: disabled


#  log4j2.properties: |
#    key = value

# Allows you to add any pipeline files in /usr/share/logstash/pipeline/
### ***warn*** there is a hardcoded logstash.conf in the image, override it first
logstashPipeline:
  logstash.conf: |
    input {
      http {
        host => "0.0.0.0"
        port => "8080"
      }
    }
    
    output {
      opentelemetry {
        endpoint => "http://10.72.1.12:4318/v1/logs"
        protocol => "http"
        body => "[log]"
        severity_text => "[level]"
        compression => "gzip"
      }
    }

logstashJavaOpts: >-
  -Xmx4g -Xms4g
  -javaagent:/opt/dd-java/dd-java-agent.jar
  -Ddd.profiling.enabled=true
  -Ddd.profiling.ddprof.liveheap.enabled=true
  -Ddd.profiling.ddprof.wall.enabled=true

resources:
  requests:
    cpu: "4000m"
    memory: "8Gi"
  limits:
    cpu: "4000m"
    memory: "8Gi"

high lock contention

datadog said 75% of cpu is cost in lock contention.

[zdyj3170101136]

log receive 3w/s。

logstash 2 pod, 2 core cpu usaeg, 4.5 GB memory.