Cannot make trino 2-node cluster work #11854

[arunmarathe]

Please refer to the closed issue number 11854. I cannot reopen that issue, and hence opening a new one.

Here is the config.properties file for the coordinator.

[arunm@vm-arunm etc]$ cat config.properties
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
query.max-memory=5GB
query.max-memory-per-node=1GB
discovery.uri=http://localhost:8080
discovery-server.enabled=true

Here is the config.properties file for the only worker.

coordinator=false
http-server.http.port=8080
query.max-memory=5GB
query.max-memory-per-node=1GB
discovery.uri=http://IP-addr-of-the-coordinator:8080

I can ping the coordinator node from the worker node. Both are CentOS.

Still even the following simple query hangs.

trino> SELECT * FROM system.runtime.nodes;

Query 20220418_193646_00000_b4bes, QUEUED, 0 nodes, 0 splits

Can someone help? Thanks.

Arun

[ebyhr]

Could you share the below information on coordinator and workers?

• node.properties
• jvm.config
• server log

[zhaoyim]

@arunmarathe could you use the hostname or IP instead the localhost in discovery.uri=http://localhost:8080 and make sure the port 8080 are not used by other APP, or change to another port Thanks!

[arunmarathe]

@arunmarathe could you use the hostname or IP instead the localhost in discovery.uri=http://localhost:8080 and make sure the port 8080 are not used by other APP, or change to another port Thanks!

I used IP address in discovery.uri, and changed the port to 8099. Neither of them helped.
The query I posted still hangs.

[arunmarathe]

Could you share the below information on coordinator and workers?

• node.properties
• jvm.config
• server log

For coordinator:

node.properties:

node.environment=demo
node.id=coordinator1

jvm.config:

-server
-Xmx4G
-XX:-UseBiasedLocking
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
-XX:ReservedCodeCacheSize=512M
-XX:PerMethodRecompilationCutoff=10000
-XX:PerBytecodeRecompilationCutoff=10000
-Djdk.nio.maxCachedBufferSize=2000000
-Djdk.attach.allowAttachSelf=true

server log:

2022-04-19T11:06:45.704-0400    INFO    main    io.trino.exchange.ExchangeManagerRegistry       -- Loading exchange manager filesystem --
2022-04-19T11:06:46.149-0400    INFO    main    Bootstrap       PROPERTY                             DEFAULT  RUNTIME                      DESCRIPTION
2022-04-19T11:06:46.149-0400    INFO    main    Bootstrap       exchange.base-directory              ----     /tmp/trino-exchange-manager
2022-04-19T11:06:46.149-0400    INFO    main    Bootstrap       exchange.encryption-enabled          true     true
2022-04-19T11:06:46.149-0400    INFO    main    Bootstrap       exchange.sink-buffer-pool-min-size   10       10
2022-04-19T11:06:46.149-0400    INFO    main    Bootstrap       exchange.sink-buffers-per-partition  2        2
2022-04-19T11:06:46.149-0400    INFO    main    Bootstrap       exchange.source-concurrent-readers   4        4
2022-04-19T11:06:46.149-0400    INFO    main    Bootstrap       exchange.max-page-storage-size       16MB     16MB                         Max storage size of a page written to a sink, including the page itself and its size represented as an int
2022-04-19T11:06:46.264-0400    INFO    main    io.airlift.bootstrap.LifeCycleManager   Life cycle starting...
2022-04-19T11:06:46.264-0400    INFO    main    io.airlift.bootstrap.LifeCycleManager   Life cycle started
2022-04-19T11:06:46.265-0400    INFO    main    io.trino.exchange.ExchangeManagerRegistry       -- Loaded exchange manager filesystem --
2022-04-19T11:06:46.301-0400    INFO    main    io.trino.server.Server  ======== SERVER STARTED ========
2022-04-19T11:11:15.811-0400    INFO    dispatcher-query-2      io.trino.event.QueryMonitor     TIMELINE: Query 20220419_150937_00000_srwua :: Transaction:[c20385b9-e2fc-40fe-9519-4adf97bfdda6] :: elapsed 97923ms :: planning 97923ms :: waiting 0ms :: scheduling 0ms :: running 0ms :: finishing 0ms :: begin 2022-04-19T11:09:37.841-04:00 :: end 2022-04-19T11:11:15.764-04:00

For worker:

node.properties:

node.environment=demo
node.id=worker1

jvm.config:

-server
-Xmx4G
-XX:-UseBiasedLocking
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
-XX:ReservedCodeCacheSize=512M
-XX:PerMethodRecompilationCutoff=10000
-XX:PerBytecodeRecompilationCutoff=10000
-Djdk.nio.maxCachedBufferSize=2000000
-Djdk.attach.allowAttachSelf=true

server log:

2022-04-19T11:07:31.043-0400    INFO    main    io.trino.exchange.ExchangeManagerRegistry       -- Loading exchange manager filesystem --
2022-04-19T11:07:31.405-0400    INFO    main    Bootstrap       PROPERTY                             DEFAULT  RUNTIME                      DESCRIPTION
2022-04-19T11:07:31.405-0400    INFO    main    Bootstrap       exchange.base-directory              ----     /tmp/trino-exchange-manager
2022-04-19T11:07:31.405-0400    INFO    main    Bootstrap       exchange.encryption-enabled          true     true
2022-04-19T11:07:31.405-0400    INFO    main    Bootstrap       exchange.sink-buffer-pool-min-size   10       10
2022-04-19T11:07:31.405-0400    INFO    main    Bootstrap       exchange.sink-buffers-per-partition  2        2
2022-04-19T11:07:31.405-0400    INFO    main    Bootstrap       exchange.source-concurrent-readers   4        4
2022-04-19T11:07:31.405-0400    INFO    main    Bootstrap       exchange.max-page-storage-size       16MB     16MB                         Max storage size of a page written to a sink, including the page itself and its size represented as an int
2022-04-19T11:07:31.541-0400    INFO    main    io.airlift.bootstrap.LifeCycleManager   Life cycle starting...
2022-04-19T11:07:31.542-0400    INFO    main    io.airlift.bootstrap.LifeCycleManager   Life cycle started
2022-04-19T11:07:31.542-0400    INFO    main    io.trino.exchange.ExchangeManagerRegistry       -- Loaded exchange manager filesystem --
2022-04-19T11:07:31.571-0400    INFO    main    io.trino.server.Server  ======== SERVER STARTED ========

[ebyhr]

@arunmarathe By the way, could you join the community Slack so that we can debug interactively? https://trino.io/slack.html

[arunmarathe]

@arunmarathe By the way, could you join the community Slack so that we can debug interactively? https://trino.io/slack.html

I have joined the Slack channel, but am unfamiliar with how it works. Do I ask for help from a particular person?

[gianluca-moro]

Were you able to solve the issue? If yes, how?

[arunmarathe]

As far as I can remember, no. It has been a while though.