fix(rustfs): bound role probe cadence

[weicao]

Problem

In two independent RustFS beta.10 focused runs (r05 and r06), all four Pods were Ready, /health/live succeeded, roleProbe/Pod label was readwrite, and restart count stayed zero. Both runs stopped at the SB01 Cluster Running gate and did not enter S3 data-path validation.

The first role event arrived after the last InstanceSet status reconcile; instanceStatus[].role remained empty and Component/Cluster stayed Creating for 1500 seconds. This N=2 evidence proves that the role-publication control-plane chain did not converge in that exact scene. It does not yet prove cadence is the only cause, and it is not a RustFS data-path or release-wide claim.

Falsifiable addon-side candidate

• add Helm render regression tests for the RustFS roleProbe cadence and unchanged exec command;
• set periodSeconds: 1 and timeoutSeconds: 3 on lifecycleActions.roleProbe;
• keep the existing loopback health check and readwrite / notready stdout semantics unchanged.

Contract anchors:

• kubeblocks-addon-docs/docs/addon-api/05a-lifecycle-roles-and-probe.md
• kubeblocks-addon-docs/docs/addon-api/05c-lifecycle-action-contracts.md

The fresh runtime gate must separately observe role event -> Pod role label -> InstanceSet instanceStatus[].role -> Component Running -> Cluster Running. If event/label updates but InstanceSet role remains empty, the candidate is falsified and the first blocker moves to the control-plane branch.

TDD and local verification

• before implementation: the new focused ShellSpec failed with periodSeconds must be 1, got nil;
• after implementation: six focused tests pass with Bash 5.3.9;
• rendered cadence is exactly 1/3;
• canonical SHA256 of the roleProbe exec command is unchanged;
• branch tests cover wget fast success, wget fast failure + curl success, both fail, and a hanging wget consuming the 3s action budget (no role token, curl not reached);
• Helm lint and git diff --check pass.

Runtime boundary

Local checks prove chart/render and command-branch contracts only. They do not prove event scheduling, reconcile timing, CPU/throttle behavior, runtime convergence, or S3 behavior. Jean owns candidate packaging and real Kubernetes validation. r05/r06 remain diagnostic evidence only; acceptance count restarts on the new exact candidate.

Related RustFS PR audit

PR #3200 is the current separate scale-up parity guard and is intentionally not duplicated in this PR. The runtime candidate branch includes its current exact patch set separately and records both pins.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 101 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (62e3e56) to head (9d36468).

Files with missing lines	Patch %	Lines
...s/rustfs/scripts-ut-spec/roleprobe_cadence_spec.sh	0.00%	101 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##            main   #3247    +/-   ##
======================================
  Coverage   0.00%   0.00%            
======================================
  Files        152     153     +1     
  Lines      23792   23893   +101     
======================================
- Misses     23792   23893   +101     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[weicao]

Design review is clear for the exact v5 contract:

• design SHA256: 8a1bef7dfabc8ffbaed5779fe67abdb27318fe87ff2b8ec952698e6bfebf64b2
• reviewer verdict: DESIGN_CLEAR / BLOCKER_LIST 0
• product head: 9d364681b8f5ff6bab65215768f8d30b95d915e4
• local RustFS ShellSpec: 32 examples, 0 failures
• local Helm lint/render, startup bash -n, ShellCheck error-level, and diff-check: PASS
• latest current GitHub checks: PASS (the earlier label-check failure was superseded after adding nopick)

Evidence boundary: cadence remains a falsifiable addon hypothesis. Static/design evidence does not establish runtime or product PASS. Fresh focused runtime starts at N=0; a single focused PASS is scoped N=1, and a second independent same-topology sample is required to close the r05/r06-shaped issue.

[weicao]

Runtime hypothesis update — PR returned to Draft.

Tester-reported fresh r07 reached the predeclared v5 falsification terminal: exact 4/4 Pods were Ready and labeled readwrite at observer t0, while 31 consecutive 1-second successful InstanceSet API samples kept all four status.instanceStatus[].role values empty. The run reported zero API errors, zero NotFound, zero action timeouts, and zero kbagent restart delta. RustFS data-path checks were not entered.

Per the reviewed contract, this single fresh sample falsifies the claim that roleProbe cadence periodSeconds: 1 / timeoutSeconds: 3 is sufficient to fix the observed status-projection non-convergence. It does not establish a RustFS product/data-path failure. The immutable evidence pack is still being finalized and will be independently verified before Layer 4 routing.

Do not merge this PR as a runtime-validated fix. Static tests remain valid as tests of the proposed cadence behavior, but the runtime-fix hypothesis is no longer supported.

[weicao]

Independent evidence verification completed.

• Pack: beta10-focused-scale-r07-falsification-v2.tar.gz
• SHA256: d6d720701d60dc894f5364bf5b4ab3476cdc5006c10a24f6b6fdade3cd34f89d
• Archive: 210 members (206 files, 4 directories); no unsafe paths, special members, duplicate paths, or AppleDouble files
• MANIFEST: 205/205 verified
• Narrow credential-shaped scan: 0 files

The originals confirm cadence periodSeconds: 1 / timeoutSeconds: 3, exact 4/4 Pods Ready and labeled readwrite with restart delta 0, five exact controller role-event lines, and 31 consecutive successful InstanceSet API samples from 17:03:51Z through 17:04:54Z while all four roles remained empty. Component reconciliation and InstanceSet UPDATE OK records occur after the role events. The observer failed at the predeclared terminal; the main Job was suspended before OpsRequest/data-path entry.

This supports routing to the KubeBlocks control-plane InstanceSet Pod-watch/status-projection branch. It does not identify a specific controller code root cause and does not establish a RustFS data-path failure. PR remains Draft.

[weicao]

Correction: the earlier r07 runtime-falsification conclusion is withdrawn.

Controller-owner source review found that the exact live KubeBlocks v1.0.0 contract does not define status.instanceStatus[].role. In that version, role projection is represented by status.membersStatus[].role, and it is populated only when spec.roles is non-empty. The instanceStatus[].role contract was added later.

Therefore, 31 successful reads showing no instanceStatus.role are not a valid failure indicator for this exact controller version. The prior statements that cadence was runtime-falsified and that InstanceSet role status was dropped are invalidated pending a raw, unfiltered continuation that captures spec.roles, status.membersStatus, status.instanceStatus, full metadata/resource versions, Pod identities, and InstanceSet-controller logs at two points 60 seconds apart.

PR remains Draft. No second sample, workload mutation, or addon workaround will occur while the observation contract is corrected.

[weicao]

Correct-field continuation completed and independently verified.

• Pack SHA256: a2b6f0c7a15553f8b779669ae1d86cc2bc14dd88157a6d29f0262a7772ac724c
• Archive: 25 members (24 files, 1 directory); no unsafe paths/types, duplicates, AppleDouble, or narrow credential-shaped matches
• MANIFEST: 23/23 verified
• Seven read-only kubectl commands: rc=0

Two raw, unfiltered InstanceSet snapshots were taken about 62 seconds apart. Both have the same UID 03ed8bd0-4e6d-4256-92f2-1a4a2e786da8, generation 1, and resourceVersion 78641. In both snapshots, spec.roles contains readwrite and notready, while status.membersStatus is empty. The four raw Pod objects are stable, Ready, labeled readwrite, have zero restarts in both containers, and are owned by that exact InstanceSet.

This satisfies the controller owner's correct v1.0.0 branch for a Layer 4 candidate (spec.roles present + membersStatus empty). It does not identify a specific code root cause. The earlier wrong-field falsification remains withdrawn. Accepted runtime N remains 0; PR remains Draft while controller task #27 investigates.

[weicao]

Read-only CRD/controller continuation completed and independently verified.

• Pack SHA256: e37ad42c6f6b5e179ac15e5c2064ad97736548af346e3ef79c0ba59a293556ce
• Archive: 33 members (32 files, 1 directory); MANIFEST 31/31; no unsafe paths/types, duplicate paths, AppleDouble, or narrow credential-shaped matches
• Eleven read-only commands: rc=0; scene and suspended main Job unchanged

The live InstanceSet CRD has UID 4124226d-e117-49e0-b400-177fc486504a, resourceVersion 704. Its v1 version is served and storage, and has the status subresource, but the v1 status schema does not contain membersStatus. The same CRD's served, non-storage v1alpha1 schema does contain the complete membersStatus field.

The manager log archive contains 1,068 raw lines. The eight exact controllerKind=InstanceSet lines are only controller/event-source/worker startup lines (including a Pod source), with no target-object reconcile line. The InstanceSet and four Pods remain byte-stable in the relevant identity/status fields.

Current branch: live CRD/installation drift candidate. Specific root cause remains undetermined and no CRD patch or reconcile trigger is authorized. Accepted runtime N remains 0; PR stays Draft.

[weicao]

Controller-owner terminal classification: Layer 3 controller/CRD version skew in this test environment.

The live manager is the official KubeBlocks v1.0.0 binary. That source contract writes roles to status.membersStatus[]; its old instanceStatus[] structure has no role field. The live served/storage v1 CRD is from the post-#9719 contract instead: it removes membersStatus and defines instanceStatus.items.role. The live canonical v1 schema matches the post-#9719 source and differs from the v1.0.0 release schema.

This explains the observations directly: the old controller's membersStatus write is pruned by the live v1 structural schema, while its old instanceStatus contains only pod names/configs and cannot populate the newer role field. Startup warnings for additional unknown old-binary fields show the skew is broader than one status field.

Scope: one directly observed environment/scene. This is sufficient as that environment's first blocker, but it is not generalized to other environments or releases. It is not a RustFS/addon/data-path bug and not a new v1.0.0 Pod-watch defect.

Disposition: keep this PR Draft. r07 is N=0 and cannot validate the cadence change. The environment must align the manager image and the complete KubeBlocks CRD set to one exact release/commit; do not patch a single CRD field or add an addon workaround. A fresh, version-consistent scene is required before role-projection N=1 can be evaluated.

[weicao]

Environment alignment gate is now independently accepted, but this does not add a runtime sample.

Evidence pack: Raft attachment 247ea1dd-eb89-4a14-b798-e3f712be6302, SHA-256 b0634bdb2fa7b58cb6ec510169e5bd5f129b1065411d057121d699b315b6827d.

Independent checks:

• archive: 138 members; MANIFEST 136/136; unsafe path/type and narrow credential-shaped matches: 0
• command ledger: 66/66 rc=0
• complete CRD set: 31/31 Established; server-side dry-run-normalized .spec hashes match live 31/31, mismatch=0
• InstanceSet v1 schema contains both membersStatus and instanceStatus
• manager identity is exact kubeblocks:1.0.0@sha256:cf6e485a…50a08, Ready, restart=0; DataProtection Ready/restart=0; workload CR count=0

Scope: ALIGN_PASS closes only the controller/CRD version-consistency environment gate. RustFS/roleProbe runtime acceptance remains N=0. This PR stays Draft until a fresh isolated role-projection focused scene produces a bounded result; that result will be counted only as N=1 and will not reuse r07.