trino in docker

[shilin-ai]

How to connect trino-coordinator and trino-workers it they are running on different docker hosts?
Trino-worker try to connect to coordinator, but it use internal docker ip to "identify", then there are such errors in logs:

2022-12-14T08:05:43.543Z	WARN	http-client-node-manager-281	io.trino.metadata.RemoteNodeState	Error fetching node state from http://10.13.0.10:26309/v1/info/state: Server refused connection: http://10.13.0.10:26309/v1/info/state
2022-12-14T08:05:48.544Z	WARN	http-client-node-manager-347	io.trino.metadata.RemoteNodeState	Error fetching node state from http://10.13.0.10:26309/v1/info/state: Server refused connection: http://10.13.0.10:26309/v1/info/state
2022-12-14T08:05:53.545Z	WARN	http-client-node-manager-390	io.trino.metadata.RemoteNodeState	Error fetching node state from http://10.13.0.10:26309/v1/info/state: Server refused connection: http://10.13.0.10:26309/v1/info/state
2022-12-14T08:05:58.546Z	WARN	http-client-node-manager-356	io.trino.metadata.RemoteNodeState	Error fetching node state from http://10.13.0.10:26309/v1/info/state: Server refused connection: http://10.13.0.10:26309/v1/info/state

10.13.0.10 - it is a docker network on remote machine.

[pulkitkalia1994]

Hey @shilin-ai How did you resolve this error?
I am facing the same problem

[shilin-ai]

Hi! I didn't )
So, coordinator and workers must be running on some machine, in one docker network.

[shilin-ai]

Or... docker networks must be routed between all cluster machines... if here are a few hosts, it seems not too difficult (manualy), but if here are a lot of hosts, may be with autoscalling - then i haven't ideas...

[hungtcs]

Have you tried mapping the port of the coordinator to the host, and then the workers on other docker hosts connect to the coordinator through the host ip and port?

[shilin-ai]

Yes, mapping works, but when worker "registers" at coordinator, it send its internal ip (from docker conainer).

[teoria]

using docker-compose with network_mode: bridge
works here !

[hashhar]

There are two ways of solving this. But first let's understand what's happening.

• Coordinator and workers start up.
• All nodes look at value for discovery.uri and try to register themselves to it. This URI generally points to coordinator. So this means workers must be able to reach this URI.
• When a node registers itself with the discovery service it also writes down it's own IP address so that other nodes know how to reach it.

With the above information you can see two requirements:

• discovery.uri MUST be reachable from all nodes.
  • This can be done by for example assigning a public IP to coordinator and using that as discovery.uri or using a URI that is routable from workers etc. etc.
• Each worker must be reachable from coordinator.
  • This can be done in two ways:
    • Workers live in same network as coordinator.
    • Workers send out an externally routable address to discovery service. This can be done by setting either of the two properties according to your infrastructure:
      • https://github.com/airlift/airlift/blob/9ba31f23d885c63a6c0b198aeacf4e7aebe919ca/node/src/main/java/io/airlift/node/NodeConfig.java#L177
      • https://github.com/airlift/airlift/blob/9ba31f23d885c63a6c0b198aeacf4e7aebe919ca/node/src/main/java/io/airlift/node/NodeConfig.java#L121

---

For this specific case it's simply easier to not use docker networking and instead use the underlying host network.
If you really want to use docker networking then you need to setup overlay networks like described in https://docs.docker.com/engine/network/tutorials/overlay/.

[samer1977]

Hi ,
Not only worker needs to be reachable from coordinator but it seems like worker need to reach each other! which I did not expect.
I'm trying to use docker windows desktop to deploy multi host trino cluster . As you know windows docker doesnt work well with host network so using the host IP is not an option. I solved the problem of worker-cooredinator communication using Reverse Proxy, this work well if you have one worker and one coordinator ( which can be a worker as well) , however once I have added an an additional worker, I started getting an error on each worker stating that its not able to reach the other (since each docker container IP is being sent over) . I still can solve this by adding more Reverse Proxy but its going to complicate things even further. I'm not sure Nginx Reverse Proxy can mask the host IP to the remote Ip so that coordinator\worker will see the other worker host IP but I could not figure this out and Im not sure if its even possible. Does anyone have other thoughts or suggestions? Where do I set the variable in mentioned above links and which one do I use (node.internal-address-source or node.external-address)? They dont seem to be related in any way to Tirno.