[JS2] Understand what it takes to upgrade a JS2 cluster's k8s version

[GeorgianaElena]

Context

2i2c's JS2 cluster is on K8s 1.31.

What we need to do

We need to upgrade. However, we never did this and don't have docs. Upgrading might actually mean re-creating the cluster.

Relevant issues:

• https://github.com/2i2c-org/meta/issues/2917
• [Jetstream2] k8s version upgrades concern #6464

Definition of Done

• 2i2c's JS2 cluster runs 1.34
• We understand if it's possible and update the infra appropriately so that upgrading doesn't mean cluster recreation
• Docs are created

[julianpistorius]

Hi @GeorgianaElena! No, you do not have to recreate the cluster. The Magnum API provides a way to upgrade a cluster in place:

https://docs.openstack.org/api-ref/container-infrastructure-management/#upgrade-a-cluster

Via the CLI:

https://docs.openstack.org/python-openstackclient/latest/cli/plugin-commands/magnum.html#coe-cluster-upgrade

usage: openstack coe cluster upgrade [-h] [--max-batch-size <max_batch_size>] [--nodegroup <nodegroup>]
                                     <cluster> cluster_template

This operation results in a rolling upgrade of the cluster which replaces existing nodes with new nodes, <max_batch_size> nodes at a time.

Note: As usual, do not skip Kubernetes versions. Your cluster will end up an error state if you do (though it is recoverable).

To upgrade from kubernetes-1-30-jammy to kubernetes-1-33-jammy I'd run the following commands:

openstack coe cluster upgrade my-cluster kubernetes-1-31-jammy
# Wait until cluster is upgraded to Kubernetes 1.31
openstack coe cluster upgrade my-cluster kubernetes-1-32-jammy
# Wait until cluster is upgraded to Kubernetes 1.32
openstack coe cluster upgrade my-cluster kubernetes-1-33-jammy

Hope this helps. Let me know if you need any more information.

[julianpistorius]

This is also good timing. I was going to reach out to you this week about upgrading your clusters in the next month.

Jetstream2 uses the Cluster API driver for Magnum, and clusters created using this driver need to be upgraded every 12 months (or more frequently), otherwise the authentication certificates will expire and you will lose access to the Kubernetes control planes. Once upgraded you have to retrieve new Kubernetes configuration details for calling the Kubernetes API. More information:

• https://gitlab.com/jetstream-cloud/jetstream2/project-management/-/issues/345
• Does cluster-api renew certificates of workload cluster? kubernetes-sigs/cluster-api#5435

You have some of the oldest clusters in Jetstream2 - almost 11 months old (2025-02-28) - so you have about 5 weeks left to do this.

Let me know if you need more information.

[julianpistorius]

Hello again @GeorgianaElena! We should have Kubernetes 1.34 available soon (like in a week or so).

[GeorgianaElena]

Hey @julianpistorius! Thank you very much for chiming in here. This is extremely valuable info.

We hope to get to this upgrade in early February. We might come back for help then, if we run into issues or have questions.
Hope that's ok.

[agoose77]

It looks like our cluster is stuck in UPDATE_FAILED. I'm not sure how this happened — any of us could have been working on this cluster! But, I think we need to recreate the cluster at this point.