How do trino clusters rolling upgrade

[akenO8]

I want to do a rolling upgrade of the trino cluster without affecting production services.
Does trino support different versions of nodes?

When I first upgrade a woker, the woker status becomes inactive.

When I upgrade coordinator first, the status of all woker nodes becomes inactive, as shown in the following figure

This is not the effect I want. Is there a better way?
Thanks for your advice.

[hashhar]

Trino requires all nodes to have same version and environment.

So you cannot rolling restart a cluster without workers going offline. Most people do a blue-green instead.

[hashhar]

An alternative is to launch new coordinator with 417. And then rolling upgrade the workers so that they join 417 cluster instead of remaining in 416 cluster.

[akenO8]

Thank you.
Well, that's one solution, by temporarily adding an extra coordinator.
However, coordinator does not support graceful shutdown, so when the old(416) coordinator are being upgraded, services are still affected.

[hashhar]

At some previous place we used DNS address for clients to connect to coordinator. After new cluster was up and we had validated it (ran some test workloads directly against new coordinator address) we would update DNS to point to new coordinator.

This worked because any client who executed query on old cluster got the older cluster coordinator address in the responses and continued reading results from there. After some time when we noticed 0 queries on old coordinator we removed it.

So you can do:

Current cluster:
DNS - mytrino.mycompany.com resolves to C-old
Coordinator - C-old
Workers - W1-old...Wn-old
configure min-required-workers as n/2 for example

• Launch new coordinator C-new

• For each worker:

  • Send graceful shutdown request
  • Once worker is stopped upgrade that worker to new version Wn-new

• Once new cluster has at-least min-required-workers switch DNS to point to C-new

  • Now all new queries will go to new cluster, old ones will keep executing on old cluster

• For all remaining workers on old version

  • Send graceful shutdown request
  • Once worker is stopped upgrade that worker to new version Wn-new

• By now all workers are upgraded to new version and old cluster has 0 running queries

• Drop the old coordinator

• Upgrade is finished

[akenO8]

When I switch DNS to point to C-new,
Now all new queries will go to new cluster, old ones will keep executing on old cluster
But how does the client get the result of the old query?

[hashhar]

the coordinator doesn't know DNS name, it just knows it's own URL. So in all response it sends to clients the URL is it's direct URL - not via DNS. So clients can keep following the URL and always reach same coordinator.

This is how the client protocol is designed. Clients just follow the nextUri that coordinator gives them and coordinator will always point that to itself so only the first connection per query needs DNS, future connections don't.

[akenO8]

Thank you very much for your answer. I understand.