Upgrade nebari config without upgrading Kubernetes version

[shauryagoel]

I upgraded K8s version of nodes in Nebari (on GKE) a month back. Now, I want to update some configuration parameters in nebari-config.yaml like node resources in jupyter hub profile. However, on rendering the changes, I am seeing the following error-

Invalid `kubernetes-version` provided: 1.30.9-gke.1009000.
Please select from one of the following supported Kubernetes versions: ['1.31.6-gke.1140000', ... ] or omit flag to use latest Kubernetes version available. [type=value_error, input_value={'kubernetes_version': '1...ject': 'data'}, input_type=CommentedMap]
    For further information visit https://errors.pydantic.dev/2.9/v/value_error

On running the gcloud container get-server-config --region asia-south1 command, I can see that the version 1.30.9-gke.1009000 is not in any of the RAPID, REGULAR or STABLE channels. But I can see it in the validNodeVersions section.
How can I avoid this without upgrading the K8s version? This will become problematic in future if I want to upgrade nebari config again in a few days, then, it may ask for K8s version upgrade as well. How can I prevent this?

[marcelovilla]

Hey @shauryagoel, can you try specifying 1.30.9 instead of 1.30.9-gke.1009000 in your nebari-config.yaml file? This should avoid Nebari raising a validation error, but I'm not 100% sure it would stop the cluster from upgrading to a newer GKE patch version.

Let me know if this helps.

[shauryagoel]

I tried using 1.30.9, however, nebari/terraform upgraded the node version to v1.30.9-gke.1231000 without confirming with me. Even though I had set release_channel to UNSPECIFIED.

[marcelovilla]

This is definitely not the expected behavior. It might have to do with the channel we're using to pull the GKE versions or with the way GKE updates their available GKE patches. We'll investigate this and see if there's a way of addressing it from Nebari.

@dcmcand do you have any ideas?

[dcmcand]

We could think about adding an option to skip the k8s version check, then you could pin the gke k8s version even if it wasn't listed in the release channels.

This seems to only be an issue with gke, since they have additional releases with the same major, minor, and patch versions. So another answer might be to just deploy to aws or azure if this behavior is a problem as I think they both just use the straight semver k8s version.

[shauryagoel]

Yes, having an option to skip the K8s version would definitely help.

[viniciusdc]

I agree with what @marcelovilla and @dcmcand have already mentioned about possible approaches. The main reason why you end up getting these updates here is that when Terraform (OpenTofu) remains pinned to a specific version—like 1.31.6—it ends up conflicting with GKE’s auto-updates and eventually tries to “correct” the node pool internal local state back to the provided version.

You can choose any currently available patch version GKE supports from Nebari's CLI API response for a short-term workaround and run a few deployments. However, this only prevents the same issue from happening again since that version will eventually be replaced by a newer patch under GCP automatically.

For the issue of the missing version error you first saw, If you are willing to dig deep, I remember inspecting the API responses from our validation schemas

nebari/src/_nebari/provider/cloud/amazon_web_services.py

Lines 88 to 104 in 0a932d4

	def kubernetes_versions(region: str) -> List[str]:
	"""Return list of available kubernetes supported by cloud provider. Sorted from oldest to latest."""
	# AWS SDK (boto3) currently doesn't offer an intuitive way to list available kubernetes version. This implementation grabs kubernetes versions for specific EKS addons. It will therefore always be (at the very least) a subset of all kubernetes versions still supported by AWS.
	session = aws_session(region=region)
	client = session.client("eks")
	supported_kubernetes_versions = list()
	available_addons = client.describe_addon_versions()
	for addon in available_addons.get("addons", None):
	for eksbuild in addon.get("addonVersions", None):
	for k8sversion in eksbuild.get("compatibilities", None):
	supported_kubernetes_versions.append(
	k8sversion.get("clusterVersion", None)
	)
	supported_kubernetes_versions = sorted(list(set(supported_kubernetes_versions)))
	return filter_by_highest_supported_k8s_version(supported_kubernetes_versions)

and asserted that all the expected versions were returning, but something was removing them in the subsequent parsing/loop. Is the filtering removing them?

Overall, we encourage users to use only major/minor versions (no exact patch); this way, once GKE manages patch upgrades automatically, Terraform has fewer mismatches during apply.