Trino Error: "nodes is empty" - java.lang.IllegalArgumentException: nodes is empty

[xhuynx]

I'm encountering a java.lang.IllegalArgumentException with the message "nodes is empty" in Trino when running SQL queries with 2 worker.
My sample sql 'ALTER TABLE table EXECUTE optimize(file_size_threshold => '512MB')'
or insert statement.

Note: If the query run on single node, it works fine. but If run on over 2 nodes, it will be error.
Now I stopped worker and run on 1 node on coordinator.

Can anyone help me for this issue? Thank in advance.

The error logs:

java.lang.IllegalArgumentException: nodes is empty
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at io.trino.sql.planner.NodePartitioningManager.createArbitraryBucketToNode(NodePartitioningManager.java:357)
at io.trino.sql.planner.NodePartitioningManager.getBucketNodeMap(NodePartitioningManager.java:255)
at io.trino.sql.planner.NodePartitioningManager.getBucketCount(NodePartitioningManager.java:239)
at io.trino.operator.exchange.LocalExchange.getBucketCount(LocalExchange.java:292)
at io.trino.operator.exchange.LocalExchange.createPartitionFunction(LocalExchange.java:265)
at io.trino.operator.exchange.LocalExchange.lambda$new$15(LocalExchange.java:183)
at io.trino.operator.exchange.LocalExchange.createSink(LocalExchange.java:336)
at io.trino.operator.exchange.LocalExchange$LocalExchangeSinkFactory.createSink(LocalExchange.java:461)
at io.trino.operator.exchange.LocalExchangeSinkOperator$LocalExchangeSinkOperatorFactory.createOperator(LocalExchangeSinkOperator.java:57)
at io.trino.operator.DriverFactory.createDriver(DriverFactory.java:107)
at io.trino.execution.SqlTaskExecution$DriverSplitRunnerFactory.createDriver(SqlTaskExecution.java:667)
at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:878)
at io.trino.execution.executor.dedicated.SplitProcessor.run(SplitProcessor.java:77)
at io.trino.execution.executor.dedicated.TaskEntry$VersionEmbedderBridge.lambda$run$0(TaskEntry.java:201)
at io.trino.$gen.Trino_476____20250619_045505_2.run(Unknown Source)
at io.trino.execution.executor.dedicated.TaskEntry$VersionEmbedderBridge.run(TaskEntry.java:202)
at io.trino.execution.executor.scheduler.FairScheduler.runTask(FairScheduler.java:177)
at io.trino.execution.executor.scheduler.FairScheduler.lambda$submit$0(FairScheduler.java:164)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:128)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:80)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
at java.base/java.lang.Thread.run(Thread.java:1447)

[Trifonov-Artem]

same error

ALTER TABLE iceberg_minio.test.partition_test EXECUTE optimize;

io.trino.event.QueryMonitor TIMELINE: Query 20250730_043701_00099_cup5t :: FAILED (GENERIC_INTERNAL_ERROR)

Trino_476

[ebyhr]

Did you deploy catalog config properties to workers? Does Web UI show the workers correctly?

[xhuynx]

Yes. I had deploy catalogs to worker and in webui show correct worker. I can view worker information in webui, and in query it show both worker and coordinator, but it failed on worker. Vào 07:25 PM T.4, 30 Th7, 2025 Yuya Ebihara ***@***.***> đã viết:

…

Did you deploy catalog config properties to workers? Does Web UI show the workers correctly? — Reply to this email directly, view it on GitHub <#26034 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BPSMJRJOIP24QPIWB2ZW6C33LC2U5AVCNFSM6AAAAAB7U54BFWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGOJTGQYTCOI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ebyhr]

Could you share the entire steps to reproduce? e.g. catalog config propertes, CREATE TABLE, INSERT, OPTIMIZE statements

[xhuynx]

I have been using this config for a long time from 43x - 45x. Now I implement it on docker swarm, both coordinator and workers run in the same network but different hosts. After deploying, the connection between coordinator and workers is fine, I can see information in Web UI, I can create schema, or execute light queries if running on only 1 worker or coordinator. But when executing query that requires 2 or more nodes, this error arises. Everything work fine in 1 trino server both for coordinator and worker.

My config:

Jvm.config

-server
-Xms24G
-Xmx24G
-XX:InitialRAMPercentage=10
-XX:MaxRAMPercentage=80
-XX:G1HeapRegionSize=32M
-XX:+ExplicitGCInvokesConcurrent
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:-OmitStackTraceInFastThrow
-XX:ReservedCodeCacheSize=512M
-XX:PerMethodRecompilationCutoff=10000
-XX:PerBytecodeRecompilationCutoff=10000
-Djdk.attach.allowAttachSelf=true
-Djdk.nio.maxCachedBufferSize=2000000
-Dfile.encoding=UTF-8
-XX:+EnableDynamicAgentLoading
-XX:+UnlockDiagnosticVMOptions
-XX:G1NumCollectionsKeepPinned=10000000
-XX:+EnableDynamicAgentLoading
-Xlog:gc*,safepoint::time,level,tags,tid
-agentpath:/etc/trino/bin/libjvmkill.so
-Djava.security.egd=file:/dev/urandom

config.properties
-- in worker use the same file just change coordinator = false, node-scheduler.include-coordinator=false

coordinator=true
node-scheduler.include-coordinator=true
discovery.uri=https://coordinator:8443
http-server.process-forwarded=true
http-server.authentication.type=PASSWORD
password-authenticator.config-files=${ENV:TRINO_HOME}/etc/pwddb.properties, ${ENV:TRINO_HOME}/etc/ldap.properties
http-server.https.enabled=true
http-server.https.port=8443
internal-communication.https.required=true
internal-communication.shared-secret=${ENV:TRINO_SECRET}
http-server.authentication.allow-insecure-over-http=true
http-server.https.secure-random-algorithm=SHA1PRNG

iceberg catalog

connector.name=iceberg
hive.metastore.uri=${ENV:MTS_URL}
hive.metastore.thrift.delete-files-on-drop=true
s3.endpoint=${ENV:S3_ENDPOINT}
s3.path-style-access=true
s3.aws-access-key=${ENV:S3_ACCESS_KEY}
s3.aws-secret-key=${ENV:S3_SECRET_KEY}
s3.region=us-east-1
fs.native-s3.enabled=true
iceberg.remove-orphan-files.min-retention=60s
iceberg.expire-snapshots.min-retention=60s
iceberg.register-table-procedure.enabled=true
iceberg.add-files-procedure.enabled=true

[Trifonov-Artem]

I turned off node-scheduler.include-coordinator=false and now it works for me

[xhuynx]

You right, node-scheduler.include-coordinator has default value = true. I was removed it in my worker config.properties.