Skip to content

✨ Cancel instance refresh on any relevant change to ASG instead of blocking until previous one is finished (which may have led to failing nodes due to outdated join token) #5543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

✨ Cancel instance refresh on any relevant change to ASG instead of blocking until previous one is finished (which may have led to failing nodes due to outdated join token) #5543

wants to merge 1 commit into from
+236 −108

Conversation

AndiDog
Copy link
Contributor

@AndiDog AndiDog commented Jun 11, 2025

(reopened from #5173 because I didn't have enough permissions to force-push)

What type of PR is this?

What this PR does / why we need it:

Changing any relevant spec.* for an AWSMachinePool triggers rolling of nodes via ASG instance refresh. If another change happens shortly afterwards, it has to wait until the first rollout is done, and will then trigger another instance refresh. But it is neither necessary nor desired to roll all worker nodes twice in such a case, and it's much slower. Instead, cancel the first pending instance refresh, wait until another one can be started, and apply the latest change as soon as possible with the second instance refresh.

This change has been running fine in Giant Swarm's CAPA fork for almost a year at the time of opening this PR.

Checklist:

  • squashed commits
  • includes documentation
  • includes emoji in title
  • adds unit tests
  • adds or updates e2e tests

Release note:

Cancel instance refresh on any relevant change to ASG instead of blocking until previous one is finished

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 11, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dlipovetsky for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added needs-priority size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 11, 2025
@AndiDog AndiDog force-pushed the cancel-instance-refresh branch from 56f83a4 to a99b8ca Compare June 11, 2025 15:23
@AndiDog
Copy link
Contributor Author

AndiDog commented Jun 12, 2025

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

@AndiDog AndiDog force-pushed the cancel-instance-refresh branch from a99b8ca to af59d86 Compare June 24, 2025 07:02
@AndiDog AndiDog force-pushed the cancel-instance-refresh branch 2 times, most recently from 615c57c to d4d3662 Compare July 2, 2025 14:41
@AndiDog
Copy link
Contributor Author

AndiDog commented Jul 16, 2025

@richardcase I think you have some machine pool experience and could maybe review this?

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 29, 2025
@AndiDog
Cancel instance refresh on any relevant change to ASG instead of bloc…
baf3527 
…king until previous one is finished (which may have led to failing nodes due to outdated join token)
@AndiDog AndiDog force-pushed the cancel-instance-refresh branch from d4d3662 to baf3527 Compare August 28, 2025 14:15
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 28, 2025
@AndiDog
Copy link
Contributor Author

AndiDog commented Aug 28, 2025

Rebased onto AWS SDK Go v2 changes

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

@AndiDog
Copy link
Contributor Author

AndiDog commented Aug 28, 2025

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

@AndiDog
Copy link
Contributor Author

AndiDog commented Aug 30, 2025

Rate limit errors in the test, trying again

/test pull-cluster-api-provider-aws-e2e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@damdo damdo Awaiting requested review from damdo

@richardcase richardcase Awaiting requested review from richardcase

Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AndiDog @k8s-ci-robot