🐛 Fix race conditions ScaleDownOldMS #12812

fabriziopandini · 2025-10-01T11:46:55Z

What this PR does / why we need it:
This PR fixes two race conditions in the logic that is responsible for scaling down OldMSs when performing rollingRollout.
The two race conditions were surfaced and documented in the context of the work for #12804. More specifically:

fails when the MD controller is called twice in a row, without MS controller being triggered in the between, e.g.
- first reconcile scales down ms1, 6-->5 (-1)
- second reconcile is not taking into account scales down already in progress, unhealthy count is wrongly computed as -1 instead of 0, this leads to increasing replica count instead of keeping it as it is (or scaling down), and then the safeguard below errors out.
fails when the MD controller is called twice in a row e.g. reconcile of md 6 replicas MaxSurge=3, MaxUnavailable=1
- when current state is: ms1, 6/5 replicas << one is scaling down, but scale down not yet processed by the MS controller, ms2, 3/3 replicas
- reconcile leads to: ms1, 6/1 replicas << it further scaled down by 4, which leads to totAvailable machines is less than MinUnavailable, which should not happen

Notably after the fix:

All the rollout sequence tests with default rollout order are completed without any change from the old logic
It is now possible to run rollout sequence tests with random rollout order, increasing the number of tested scenarios for 9 to 918

Which issue(s) this PR fixes *(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the Part of #12291

/area machinedeployment

stmcginnis

Code wise everything looks fine to me. I'm being a little nit picky on some docstring comments, just thinking from the context of reading through this code at some point in the future needing to understand what is happening once the current context is lost.

stmcginnis · 2025-10-03T15:49:38Z

internal/controllers/machinedeployment/machinedeployment_rolling.go

+	// NOTE: we are scaling up unavailable machines first in order to increase chances for the rollout to progress;
+	// however, the MS controller might have different opinion on which machines to scale down.
+	// As a consequence, the scale down operation must continuously assess if reducing the number of replicas
+	// for an older MS could further impact availability under the assumption than any scale down could further impact availability (same as above).


I'm having a hard time parsing this sentence.

Rephrased + added an example, PTAL

stmcginnis · 2025-10-03T16:21:32Z

internal/controllers/machinedeployment/machinedeployment_rolling.go

+	// Then scale down old MS up to zero replicas / up to residual totalScaleDownCount.
+	// NOTE: also in this case, continuously assess if reducing the number of replicase could further impact availability,


Also having trouble parsing this sentence too. I think you're saying we will scale down the MS decrementing by totalScaleDownCount without going below 0?

Suggested change

// Then scale down old MS up to zero replicas / up to residual totalScaleDownCount.

// NOTE: also in this case, continuously assess if reducing the number of replicase could further impact availability,

// Then scale down old MS by totalScaleDownCount to get to zero.

// NOTE: also in this case, continuously assess if reducing the number of replicas could further impact availability,

internal/controllers/machinedeployment/machinedeployment_rolling.go

fabriziopandini · 2025-10-06T20:35:33Z

@sbueringer @stmcginnis thanks for the first round of feedback,
I have reviewed the PR to add examples, improve godoc + also reviewed part of the logic in scaleDownOldMSs
PTAL

fabriziopandini · 2025-10-06T20:35:58Z

/test pull-cluster-api-e2e-main

sbueringer · 2025-10-07T07:43:20Z

Reviewing now. I'm also going to open a PR against your PR to fixup some minor doc findings

internal/controllers/machinedeployment/machinedeployment_rolling.go

internal/controllers/machinedeployment/machinedeployment_rolling_test.go

internal/controllers/machinedeployment/machinedeployment_rollout_sequence_test.go

Signed-off-by: Stefan Büringer [email protected]

sbueringer · 2025-10-07T09:54:49Z

PR with minor fixups. It should not contain any logic changes (just godoc / variable renames / etc), but please double check: fabriziopandini#314

🌱 Fix ScaleDownOldMS: Fix review findings

internal/controllers/machinedeployment/machinedeployment_rolling_test.go

sbueringer · 2025-10-07T15:25:24Z

Thank you!

/lgtm
/approve

/hold
for other reviews

k8s-ci-robot · 2025-10-07T15:25:32Z

LGTM label has been added.

Git tree hash: 45c79a98bd7773b41dc97d99dcab2d8b613ecdac

k8s-ci-robot · 2025-10-07T15:25:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sbueringer · 2025-10-07T15:27:52Z

/test pull-cluster-api-e2e-main

Fix ScaleDownOldMS

d16533a

k8s-ci-robot added area/machinedeployment Issues or PRs related to machinedeployments cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 1, 2025

k8s-ci-robot requested review from chrischdi and JoelSpeed October 1, 2025 11:47

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 1, 2025

fabriziopandini mentioned this pull request Oct 1, 2025

Tracking issue for In-place Updates implementation #12291

Open

47 tasks

stmcginnis reviewed Oct 3, 2025

View reviewed changes

Minor improvements

142d1de

fabriziopandini added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Oct 4, 2025

fabriziopandini mentioned this pull request Oct 6, 2025

OnDelete rollout strategy produces negative replica counts causing infinite reconciliation loop #12815

Open

sbueringer reviewed Oct 6, 2025

View reviewed changes

Address feedback

aca5de4

sbueringer reviewed Oct 7, 2025

View reviewed changes

Fix review findings

7e8ac3a

Signed-off-by: Stefan Büringer [email protected]

Merge pull request #314 from sbueringer/fix-scaleDownOldMS-fixups

26e82c4

🌱 Fix ScaleDownOldMS: Fix review findings

sbueringer reviewed Oct 7, 2025

View reviewed changes

internal/controllers/machinedeployment/machinedeployment_rolling_test.go Outdated Show resolved Hide resolved

internal/controllers/machinedeployment/machinedeployment_rolling_test.go Show resolved Hide resolved

More comments

40f23d9

fabriziopandini force-pushed the fix-scaleDownOldMS branch from 151b53a to 40f23d9 Compare October 7, 2025 14:54

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 7, 2025

k8s-ci-robot assigned sbueringer Oct 7, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 7, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 7, 2025

		// Then scale down old MS up to zero replicas / up to residual totalScaleDownCount.
		// NOTE: also in this case, continuously assess if reducing the number of replicase could further impact availability,

🐛 Fix race conditions ScaleDownOldMS #12812

Are you sure you want to change the base?

🐛 Fix race conditions ScaleDownOldMS #12812

Conversation

fabriziopandini commented Oct 1, 2025

Uh oh!

stmcginnis left a comment

Choose a reason for hiding this comment

Uh oh!

stmcginnis Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

fabriziopandini Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

stmcginnis Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fabriziopandini commented Oct 6, 2025

Uh oh!

fabriziopandini commented Oct 6, 2025

Uh oh!

sbueringer commented Oct 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbueringer commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbueringer commented Oct 7, 2025

Uh oh!

k8s-ci-robot commented Oct 7, 2025

Uh oh!

k8s-ci-robot commented Oct 7, 2025

Uh oh!

sbueringer commented Oct 7, 2025

Uh oh!

Uh oh!

sbueringer commented Oct 7, 2025 •

edited

Loading