Migrate AWS EKS AMI from Amazon Linux 2 to Amazon Linux 2023 #3166

Adam-D-Lewis · 2025-10-23T16:12:10Z

AWS is deprecating Amazon Linux 2 (AL2) AMIs for EKS after November 26, 2025. Kubernetes 1.32 is the last version that will support AL2 AMIs. From version 1.33 onwards, only AL2023 and Bottlerocket AMIs will be available.

This change updates the default AMI types for EKS node groups:

AL2_x86_64 → AL2023_x86_64
AL2_x86_64_GPU → AL2023_x86_64_GPU

References:

https://docs.aws.amazon.com/eks/latest/userguide/eks-ami-deprecation-faqs.html

🤖 Generated with Claude Code

Additional References

https://docs.aws.amazon.com/eks/latest/userguide/al2023.html

Reference Issues or PRs

What does this implement/fix?

Put a x in the boxes that apply

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds a feature)
Breaking change (fix or feature that would cause existing features not to work as expected)
Documentation Update
Code style update (formatting, renaming)
Refactoring (no functional changes, no API changes)
Build related changes
Other (please describe):

Documentation

For new features or enhancements, a corresponding PR has been opened in the documentation repository (if applicable)
- Link to docs PR:

Testing

Did you test the pull request locally?
Did you add new tests?

How to test this PR?

Any other comments?

AWS is deprecating Amazon Linux 2 (AL2) AMIs for EKS after November 26, 2025. Kubernetes 1.32 is the last version that will support AL2 AMIs. From version 1.33 onwards, only AL2023 and Bottlerocket AMIs will be available. This change updates the default AMI types for EKS node groups: - AL2_x86_64 → AL2023_x86_64 - AL2_x86_64_GPU → AL2023_x86_64_GPU References: - https://docs.aws.amazon.com/eks/latest/userguide/eks-ami-deprecation-faqs.html 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Adam-D-Lewis · 2025-10-23T16:14:22Z

I haven't tested this yet, but the code changes are minimal. I think we should deploy on AWS and do our standard release workflows to test.

viniciusdc · 2025-10-23T16:14:59Z

This one is a very welcoming PR, thanks @Adam-D-Lewis. I can review this once Claude finishes this. This one is imporrtante since AWS will be deprecating those AMI soon

Adam-D-Lewis · 2025-10-23T16:20:32Z

This one is a very welcoming PR, thanks @Adam-D-Lewis. I can review this once Claude finishes this. This one is imporrtante since AWS will be deprecating those AMI soon

@viniciusdc Claude is done editing. If you want to take over testing that is welcome. Thank you!

Adam-D-Lewis · 2025-10-27T20:48:38Z

The Keycloak PostgreSQL pod can't start because it's not getting volume to work with from the EBS CSI controller.

Looks like we can either set a custom launch template with http_put_response_hop_limit = 2 in the metadata_options or we can use IRSA to give the EBS CSI driver pod the needed permissions.

Relavant docs:

Adam-D-Lewis · 2025-10-27T22:34:34Z

The autoscaler is also not working for the same reason.

Relevant docs

Cluster Autoscaler not working on the new AL2023 EKS optimised AMI kubernetes/autoscaler#6963

Adam-D-Lewis · 2025-10-27T23:07:13Z

Working now, had to add a few recommended permissions. See https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md

Adam-D-Lewis · 2025-10-27T23:14:02Z

src/_nebari/stages/infrastructure/template/aws/modules/kubernetes/locals.tf

    "arn:${local.partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy",
    "arn:${local.partition}:iam::aws:policy/AmazonEKS_CNI_Policy",
-    "arn:${local.partition}:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy",
-    aws_iam_policy.worker_autoscaling.arn


the role is no longer needed on the node

Adam-D-Lewis · 2025-10-27T23:15:11Z

src/_nebari/stages/kubernetes_initialize/template/modules/cluster-autoscaler/main.tf


  repository = "https://kubernetes.github.io/autoscaler"
  chart      = "cluster-autoscaler"
-  version    = "9.19.0"


also took the opportunity to update the version here

Adam-D-Lewis · 2025-10-28T14:36:32Z

Unit Tests / Pytest (3.10) (pull_request) is failing due to a warning about the upcoming Python 3.10 EOL in a year. I've opened #3170 to mitigate that.

viniciusdc · 2025-10-28T15:11:41Z

src/_nebari/stages/infrastructure/template/aws/versions.tf

    aws = {
      source  = "hashicorp/aws"
-      version = "5.33.0"
+      version = "6.18.0"


viniciusdc · 2025-10-28T15:14:14Z

src/_nebari/stages/infrastructure/__init__.py

    if launch_template and getattr(launch_template, "ami_id", None):
        return "CUSTOM"

    if gpu_enabled:
-        return "AL2_x86_64_GPU"
+        return "AL2023_x86_64_NVIDIA"

-    return "AL2_x86_64"
+    return "AL2023_x86_64_STANDARD"




Can you check the nvidia works with those new ami's ?

Currently waiting for quota increase approval since we're on OT Nebari dev AWS account now instead of QS.

Adam-D-Lewis · 2025-10-28T15:16:22Z

@dcmcand recommends we use https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html instead of IRSA

Adam-D-Lewis · 2025-10-28T16:44:01Z

@dcmcand recommends we use https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html instead of IRSA

I looked into it a bit (including #3171). It appears that IRSA is much closer to GKE Workload Identity and Azure Workload Identity. IRSA is still fully supported and already enabled on the cluster. I'm not opposed to migrating to Pod Identity on EKS, but I think it should be done in a separate issue and not be a blocker for this PR.

Even if we switch, we'd still need to maintain IRSA capability in the short term at least. Nebari MLflow plugin (and possibly others uses it).

…MQ2aX

viniciusdc · 2025-11-13T13:46:59Z

Hey @Adam-D-Lewis thanks for working on this, what was the last status of this?

Adam-D-Lewis · 2025-11-13T15:34:43Z

It's ready for review, @viniciusdc . I have kept the IRSA approach for now for the reasons I listed above.

github-project-automation bot added this to 🪴 Nebari Project Management Oct 23, 2025

github-project-automation bot moved this to New 🚦 in 🪴 Nebari Project Management Oct 23, 2025

viniciusdc self-requested a review October 23, 2025 16:15

Adam-D-Lewis marked this pull request as ready for review October 23, 2025 16:21

Adam-D-Lewis requested a review from a team as a code owner October 23, 2025 16:21

Adam-D-Lewis requested review from dcmcand and removed request for a team October 23, 2025 16:21

correct AMI types and bump aws terraform version

25698ed

add autoscaler IRSA

8b5aa3a

Adam-D-Lewis commented Oct 27, 2025

View reviewed changes

viniciusdc reviewed Oct 28, 2025

View reviewed changes

Merge branch 'main' into claude/investigate-issue-011CUQHTM3P2xpX98sx…

2342f16

…MQ2aX

Adam-D-Lewis requested a review from viniciusdc November 18, 2025 15:07

Migrate AWS EKS AMI from Amazon Linux 2 to Amazon Linux 2023 #3166

Are you sure you want to change the base?

Migrate AWS EKS AMI from Amazon Linux 2 to Amazon Linux 2023 #3166

Uh oh!

Conversation

Adam-D-Lewis commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Additional References

Reference Issues or PRs

What does this implement/fix?

Documentation

Testing

How to test this PR?

Any other comments?

Uh oh!

Adam-D-Lewis commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viniciusdc commented Oct 23, 2025

Uh oh!

Adam-D-Lewis commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Adam-D-Lewis commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Adam-D-Lewis commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Adam-D-Lewis commented Oct 27, 2025

Uh oh!

Adam-D-Lewis Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Adam-D-Lewis Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Adam-D-Lewis commented Oct 28, 2025

Uh oh!

viniciusdc Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

viniciusdc Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Adam-D-Lewis Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Adam-D-Lewis commented Oct 28, 2025

Uh oh!

Adam-D-Lewis commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viniciusdc commented Nov 13, 2025

Uh oh!

Adam-D-Lewis commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Adam-D-Lewis commented Oct 23, 2025 •

edited

Loading

Adam-D-Lewis commented Oct 23, 2025 •

edited

Loading

Adam-D-Lewis commented Oct 23, 2025 •

edited

Loading

Adam-D-Lewis commented Oct 27, 2025 •

edited

Loading

Adam-D-Lewis commented Oct 27, 2025 •

edited

Loading

Adam-D-Lewis Oct 27, 2025 •

edited

Loading

Adam-D-Lewis commented Oct 28, 2025 •

edited

Loading