CPU is always high

[chengcxy]

running one trino(410) cluster with 3 workers in Docker.

In Test Env, 4:40 PM the cluster is slow and most of the queries are running timeout,then our product system is done.
the cpu of master node is keeping 80% and other work is less then 20%. All the servers are VM and only trino is running on these servers.

jvm.config:

-server
-Xmx10G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError

config.properties

coordinator=true
http-server.http.port=8082
query.max-memory=10GB
query.max-memory-per-node=1GB
discovery.uri=http://data5:8082
query.max-length=100000000
exchange.http-client.request-timeout=50s
task.concurrency=8
task.max-worker-threads=40
node-scheduler.max-splits-per-node=200
exchange.max-response-size=100MB
http-server.max-response-header-size=15MB
http-server.max-request-header-size=15MB
query.max-length=10000000
query.min-expire-age=500m

[hashhar]

You have very large values for task.concurrency and task.max-worker-threads. How many CPU cores does each machine actually have?

Also all o fthe CPU consuming threads seem to be doing garbage collection. You also have a very very small heap - 10GB only.

I'd recommend reducing the number of workers and instead giving each of them more memory. With such a small heap most of the time is being spent doing GC.