[Azure] [K8s] [Trino Cluster 3.7.2] Cannot connect to discovery server for announce

[nfoerster]

Hi there, recently we used the coordinator with integrated worker. However, for scaling reasons I want to separate coordinator and worker. Currently we run on Trino 3.7.2 with Kubernetes on Azure.

The workers are not able to find the discovery server of the coordinator:

2022-04-06T13:38:12.115Z    INFO    main    io.trino.server.Server    ======== SERVER STARTED ========
 2022-04-06T13:38:12.115Z    ERROR    Announcer-0    io.airlift.discovery.client.Announcer    Cannot connect to discovery server for announce
 io.airlift.discovery.client.DiscoveryException: Announcement failed with status code 404:
     at io.airlift.discovery.client.HttpDiscoveryAnnouncementClient$1.handle(HttpDiscoveryAnnouncementClient.java:104)
     at io.airlift.discovery.client.HttpDiscoveryAnnouncementClient$1.handle(HttpDiscoveryAnnouncementClient.java:97)

this is the config.properties of the coordinator:

coordinator=true
node-scheduler.include-coordinator=true
http-server.http.port=8080
http-server.authentication.type=PASSWORD
http-server.process-forwarded=true
http-server.authentication.allow-insecure-over-http=true
query.max-memory=16GB
query.max-stage-count=1000
discovery-server.enabled=true
discovery.uri=http://trino:8080/

and this for the worker:

coordinator=false
http-server.http.port=8080
http-server.authentication.allow-insecure-over-http=true
query.max-memory=16GB
query.max-stage-count=1000
discovery.uri=http://trino.trino-test:8080/

One might say, oh your discovy.uri of the worker is not the same as the one of the coordinator. Yes I changed this to the service uri where the coordinator is running, anyway same URI http://trino:8080/ does also not work.
I tested already some stuff, on coordinator pod curl http://trino:8080/ does work. On worker pod curl curl http://trino:8080/ returns 404, but curl http://trino.trino-test:8080/ succeeds.

This is really strange, so I tested the ingress to the Trino UI, which runs fine without any worker deployment. As soon as I deploy the worker deployment it shows:

HTTP ERROR 404 Not Found
URI:/ui/login.html
STATUS:404
MESSAGE:Not Found
SERVLET:org.glassfish.jersey.servlet.ServletContainer-672d8370

But WHY? It seems that the worker claim the discovery URI http://trino:8080/ and override the service, although the discovery URI differs:
2022-04-06T13:38:06.253Z INFO main Bootstrap discovery.uri ---- http://trino.trino-test:8080/

This is my service.yaml deployed:

apiVersion: v1
kind: Service
metadata:
  name: trino
spec:
  ports:
    - port: 8080
      name: trino
  selector:
    app: trino-coordinator

coordinator.yaml (part):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trino-coordinator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: trino-coordinator
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: trino-coordinator
...

worker.yaml (part):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trino-worker
spec:
  replicas: 2
  selector:
    matchLabels:
      app: trino-worker
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: trino-worker
...

Any idea whats going on? Any example k8s files available?

[pulkitkalia1994]

@nfoerster I have the same issue. Were you able to resolve it?

[nfoerster]

@pulkitkalia1994 yes we solved it, Trino Cluster runs good on k8s. We did some adaptions, but I can't remember the exact solution.

Try the following:
Coordinator: config.properties
discovery.uri=http://trino:8080

Worker: config.properties
discovery.uri=http://trino.${ENV:NAMESPACE}:8080

The rational for this is that the workers connect to the kubernetes service of trino coordinator.