Trino execution completely halts in the middle of running queries.

[pranavchaudhary3]

Hi Trino Team,
We are doing a POC with Trino cluster on EKS. I have been testing the production like workload on Trino cluster and have seen Trino completely halts in the middle of execution, as shown in the screenshot down below.
To put into contest, I am submitting 25 CTAS queries simultaneously, querying data from two connectors, namely hive and iceberg, writing to and reading from S3.
Cluster Config:
Instance Type: arm64
Trino Version: 442
1x Coordinator: 55 Cores, 450 GB Memory
20x Workers: 55 Cores, 450 GB Memory

  jvm.config:
    -server
    -agentpath:/usr/lib/trino/bin/libjvmkill.so
    -XX:InitialRAMPercentage=80
    -XX:MaxRAMPercentage=80
    -XX:+UseG1GC
    -XX:G1HeapRegionSize=32M
    -XX:+ExplicitGCInvokesConcurrent
    -XX:-HeapDumpOnOutOfMemoryError
    -XX:-ExitOnOutOfMemoryError
    -XX:-OmitStackTraceInFastThrow
    -XX:ReservedCodeCacheSize=512M
    -XX:PerMethodRecompilationCutoff=10000
    -XX:PerBytecodeRecompilationCutoff=10000
    -Djdk.attach.allowAttachSelf=true
    -Djdk.nio.maxCachedBufferSize=2000000
    -XX:+UnlockDiagnosticVMOptions
    # Reduce starvation of threads by GClocker, recommend to set about the number of cpu cores (JDK-8192647)
    -XX:GCLockerRetryAllocationCount=32

config.properties:
    http-server.http.port=8080
    query.max-memory=2500GB
    query.max-memory-per-node=120GB
    query.max-total-memory=3000GB
    query.max-concurrent-queries=20
    task.concurrency=8
    query.max-execution-time=120m
    query.max-history=1000
    query.client.timeout=120m
    query.execution-policy=phased
    node-scheduler.max-splits-per-node=200

Any advice/guidance on what is going wrong here?

[wendigo]

cc @martint

[martint]

This could be related to #21512

Try disabling the experimental.thread-per-driver-scheduler-enabled config option to see if that fixes the issue.

If that helps, can you help us get some diagnostics by setting the io.trino.execution.executor.dedicated.ThreadPerDriverTaskExecutor log level to DEBUG while the option above is enabled?