Trino is taking more time in Scheduling phase. How to optimize?

[phani-vaibhav]

Hi every one,

We have a Trino setup installed in kubernetes with 1 Coordinator & 20 Workers, with ample amount of cores and memory. What we observed is our query takes more time in scheduling phase. Also it is not consistent. Below are the logs for reference. We can see most of the cases scheduling is taking more time than the runtime. Can anyone help what could cause the spikes in scheduling phase duration?

1. TIMELINE: Query 20221128_060757_00009_78mfj :: Transaction:[808ac0ca-f631-44a2-b50e-8411e6447236] :: elapsed 4239ms :: planning 254ms :: waiting 47ms :: scheduling 2746ms :: running 1230ms :: finishing 9ms :: begin 2022-11-28T06:07:57.259Z :: end 2022-11-28T06:08:01.498Z
2. TIMELINE: Query 20221128_061401_00018_78mfj :: Transaction:[21202226-4c8d-4807-8b70-86a7eec0fad5] :: elapsed 6809ms :: planning 279ms :: waiting 78ms :: scheduling 4007ms :: running 2514ms :: finishing 9ms :: begin 2022-11-28T06:14:01.644Z :: end 2022-11-28T06:14:08.453Z
3. TIMELINE: Query 20221128_062026_00025_78mfj :: Transaction:[d85652c0-c44a-449e-8640-ed6093ba0f0c] :: elapsed 16293ms :: planning 216ms :: waiting 80ms :: scheduling 12339ms :: running 3730ms :: finishing 8ms :: begin 2022-11-28T06:20:26.253Z :: end 2022-11-28T06:20:42.546Z
4. TIMELINE: Query 20221128_062123_00026_78mfj :: Transaction:[6d37780e-6181-4d02-ab31-867ac4bf3865] :: elapsed 9584ms :: planning 217ms :: waiting 87ms :: scheduling 5978ms :: running 3378ms :: finishing 11ms :: begin 2022-11-28T06:21:23.561Z :: end 2022-11-28T06:21:33.145Z
5. TIMELINE: Query 20221128_062221_00027_78mfj :: Transaction:[4374110e-fab8-4b96-8c84-2a3bc8d021be] :: elapsed 12766ms :: planning 254ms :: waiting 80ms :: scheduling 7928ms :: running 4574ms :: finishing 10ms :: begin 2022-11-28T06:22:21.317Z :: end 2022-11-28T06:22:34.083Z
6. TIMELINE: Query 20221128_062426_00030_78mfj :: Transaction:[661553e6-9bba-4487-a5eb-4fae48bf6adc] :: elapsed 9691ms :: planning 223ms :: waiting 78ms :: scheduling 4012ms :: running 5447ms :: finishing 9ms :: begin 2022-11-28T06:24:26.306Z :: end 2022-11-28T06:24:35.997Z

Trino properties:
Coordinator:

coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
node-scheduler.max-splits-per-node=500
node-scheduler.max-pending-splits-per-task=50
http-server.http.port=8080
node-scheduler.max-splits-per-node=500
http-server.authentication.type=PASSWORD
http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/etc/certs/trino.pem
query.max-memory=2320GB
query.max-memory-per-node=100GB
discovery-server.enabled=true
discovery.uri=http://localhost:8080

Connector:

connector.name=hive-hadoop2
hive.metastore.uri=thrift://metastore:9083
hive.s3.path-style-access=true
hive.s3.endpoint=xxxxx
hive.s3.aws-access-key=xxxxx
hive.s3.aws-secret-key=xxxx
hive.non-managed-table-writes-enabled=true
hive.storage-format=ORC
hive.allow-drop-table=true
hive.allow-add-column=true
hive.allow-drop-column=true
hive.allow-drop-table=true
hive.allow-rename-table=true
hive.allow-rename-column=true
hive.recursive-directories=true
hive.ignore-absent-partitions=true
hive.orc.use-column-names=true
hive.metastore-timeout=3m
hive.metastore-refresh-interval=10m
hive.metastore-cache-ttl=60m
hive.s3.connect-timeout=30s
hive.s3.socket-timeout=30s

[findepi]

Continued at #15250