Skip to content

fix(kafka): block unsafe KRaft controller scale-in#3219

Draft
weicao wants to merge 4 commits into
audrey/pr3168-kb-comp-replicasfrom
audrey/kafka-kraft-scalein-failclosed
Draft

fix(kafka): block unsafe KRaft controller scale-in#3219
weicao wants to merge 4 commits into
audrey/pr3168-kb-comp-replicasfrom
audrey/kafka-kraft-scalein-failclosed

Conversation

@weicao

@weicao weicao commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Problem

Kafka KRaft controller membership in this addon uses a static controller.quorum.voters list. The controller-bearing ComponentDefinitions did not define a memberLeave action, so a 3-to-1 scale-in could delete two controller Pods while Kafka still expected voter IDs 0/1/2. The operation could report success even though the remaining broker could no longer complete bounded topic creation.

Solution

  • Add a deterministic, side-effect-free memberLeave failure action to kafka-combine and kafka-controller only.
  • Run the guard in the Kafka container and emit one stable diagnostic explaining the static-voter limitation.
  • Bump the chart to 1.1.0-alpha.2, including all version-derived immutable object names and references.
  • Keep separated broker scale-in outside the guard.
  • Remove unsafe controller scale-in instructions and document the supported ComponentDefinition upgrade boundary.
  • State explicitly that an OpsRequest timeout is not cancellation and that this guard does not restore replicas automatically.
  • Add source and rendered-object regression checks for lifecycle-action nesting, command/container selection, script mode and mount, immutable names/references, broker exclusion, and documentation escape paths.

Evidence

Exact candidate:

  • PR head: 67200df711766c01ec7a33829c1b0ee041b1f32a
  • chart: 1.1.0-alpha.2
  • package SHA256: 78eebdf97d1f714858ca529c2fad18d67fe180fe379fa6753179cf0a558f3146
  • KubeBlocks / DataProtection source pin: 00a42539
  • syncer source pin: b98b1ae5a
  • GitHub checks: 7/7 successful

Runtime validation on Kafka 3.3.2 is cumulative N=2 and clean for all six tested topology rows:

Topology Scale-in Expected result Runtime result
Combined 3 to 1 blocked N=2 clean
Combined 3 to 2 blocked N=2 clean
Separated controller 3 to 1 blocked N=2 clean
Separated controller 3 to 2 blocked N=2 clean
Separated broker 3 to 1 allowed N=2 clean
Separated broker 3 to 2 allowed N=2 clean

For the four protected rows, both target replicas invoked the lifecycle action and returned the fixed failure diagnostic before deletion. The OpsRequests ended Aborted; InstanceSet identity, actual replica count, Pod UIDs, KRaft voters, and exact business markers remained unchanged. No recovery patch or manual repair was used.

For the two broker-only rows, the OpsRequests succeeded, broker replicas converged to the requested count, retained broker UIDs stayed stable, controller InstanceSet/Pod identities and voters stayed unchanged, and exact post-scale business markers passed.

The old-Component upgrade path was also exercised twice: healthy 1.1.0-alpha.0 cluster -> supported ComponentDefinition upgrade -> exact 1.1.0-alpha.2 readback and business roundtrip -> combined 3-to-1 fail-closed result -> ordinary cleanup. Both runs closed cleanly.

Every accepted run used ordinary namespace cleanup to explicit NotFound, with scoped PV/Backup/Restore residue 0 and no force or finalizer modification.

Boundaries

  • This PR blocks unsafe controller scale-in; it does not implement dynamic KRaft membership.
  • It does not implement automatic rollback, cancellation, compensating scale-out, or manual repair.
  • A chart upgrade does not mutate existing Components in place. Existing clusters must use the supported ComponentDefinition upgrade workflow before relying on this guard.
  • The positive broker evidence and negative controller evidence above are scoped to the exact candidate and Kafka 3.3.2. They are not a claim for every Kafka version or every fault window.
  • A separate supported-control-path test while the Cluster was already Updating was rejected before scale-in and remains N=0. No product conclusion is drawn from that lane.
  • In-window lifecycle fault injection and Chaos Mesh fault coverage remain unavailable/N=0. This PR is not being presented as broader release readiness.
  • This PR is stacked on PR fix(kafka): use action replica env in custom ops #3202 exact head 367d32f5098a99582b22d4c72d5c28efa5e8d8df; retarget after that base is integrated.

Reviewer focus

Please focus on:

  1. whether the guard fails before any controller Pod deletion;
  2. whether only controller-bearing ComponentDefinitions receive memberLeave;
  3. whether all alpha.2 immutable names and references move together;
  4. whether the old-Component upgrade and timeout boundaries are stated accurately.

@codecov-commenter

codecov-commenter commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 85 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (367d32f) to head (67200df).

Files with missing lines Patch % Lines
.../kafka/scripts-ut-spec/kraft_scalein_guard_spec.sh 0.00% 85 Missing ⚠️
Additional details and impacted files
@@                      Coverage Diff                       @@
##           audrey/pr3168-kb-comp-replicas   #3219   +/-   ##
==============================================================
  Coverage                            0.00%   0.00%           
==============================================================
  Files                                  75      76    +1     
  Lines                                9341    9426   +85     
==============================================================
- Misses                               9341    9426   +85     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants