fix(starrocks-ce): make FE memberLeave single-shot and fail closed

[weicao]

Root cause

The FE memberLeave action treated ALTER SYSTEM DROP FOLLOWER rc=0 as completion without proving the leaving pod identity absent from a fresh membership snapshot, and the leader-removal path polled for up to 120 seconds inside a kbagent action clamped at 60 seconds. Missing target identity could also fall through to the idempotent already-absent rc0 path. These gaps could delete a Pod while leaving a ghost FE member, or time out without a classified result.

This PR is limited to the StarRocks-CE FE memberLeave product path, its CmpD timeout, and focused ShellSpec contracts.

Base and head

• Base: 41801859b685dea4f8d603b7aa4b90a7fb52e2d1
• Head: 87fda89beaaa8c218c16dc4aab29173421198a63
• Tree: be65efb11d08a5593d2fb38f9c7f0576d6b76225
• Diff: 4 files, 1000 insertions, 79 deletions

Behavior

• Requires non-empty KB_LEAVE_MEMBER_POD_NAME, FE_DISCOVERY_SERVICE_NAME, and STARROCKS_USER before runtime discovery or any MySQL/Java command. Empty passwords remain supported.
• Parses SHOW FRONTENDS as an authoritative exact 16-column snapshot with unique FE names/endpoints and exactly one leader before success or mutation.
• Matches the leaving pod by exact pod name or the pod-name. FQDN boundary, not a longer pod-name prefix.
• Uses one observation per convergence phase. A leader transfer gets one fresh snapshot; non-convergence returns classified rc1 for runtime reinvocation instead of polling in-process.
• Refreshes the leader and leaving-member endpoint after transfer before issuing DROP.
• Treats DROP rc=0 as incomplete until one fresh SHOW FRONTENDS simultaneously proves the old exact host plus EditLogPort absent and the leaving pod identity has zero matches. An endpoint change for the same pod therefore cannot produce false success.
• Classifies observed timeouts/non-convergence as retry_safe=true; missing required input, invalid snapshots, and rejected query/ALTER/BDB commands as retry_safe=false.
• Uses MYSQL_PWD rather than a password argument and redacts both credential values from diagnostics.

Time budget

• kbagent clamp: 60s
• CmpD action timeout: 50s
• Absolute internal deadline: 45s
• Default leader path command windows: query 6 + Java 7 + fresh query 6 + ALTER 6 + post-DROP query 6 = 31s TERM windows; allowing one-second kill escalation per command is 36s.
• Every MySQL/Java launch recomputes the current remaining budget inside run_bounded_command, reserves one launch-rollover second plus one --kill-after=1 second, and clips the TERM window to the remainder. If the remaining window is exhausted, no subprocess starts and the action returns classified rc124.
• GNU timeout rc124 and SIGKILL escalation rc137 both surface as rc124 with retry_safe=true.

RED to GREEN ledger

1. Empty/malformed successful snapshots could flow onward -> exact snapshot validation rejects them.
2. Duplicate FE identity or multiple leaders could be used -> fail closed before mutation.
3. A longer pod-name prefix could match the target -> exact pod/dot-FQDN boundary.
4. DROP rc0 while the exact old member remained could return success -> fresh exact-pair absence proof is mandatory.
5. DROP rc0 while the same pod remained under a new endpoint could return success -> fresh target match count must also be zero.
6. Leader transfer used a 120s loop and stale endpoint -> single fresh observation/defer and refreshed leader/target.
7. Query, ALTER, and BDB hard rejects had ambiguous retry behavior -> explicit hard-failure taxonomy with secret-safe stderr.
8. TERM/SIGKILL and stale budget rollover could overrun the action -> launch-time absolute clipping with launch/kill reserves and zero-launch exhaustion tests.
9. Missing/empty target identity could return already-removed rc0 -> required reader inputs fail before MySQL/Java/ALTER.
10. Runtime/image wiring was assumed -> exact release image pair, Bash/mysql/Java/GNU timeout/JAR, mount, and CmpD contracts.

Verification

• Focused ShellSpec: 31 examples, 0 failures
• ShellCheck: clean for production script and both specs
• bash -n: clean
• git diff --check: clean
• helm lint addons/starrocks-ce: 1 chart linted, 0 failed (icon recommendation only)
• Rendered CmpD: memberLeave.timeoutSeconds=50, /bin/bash -c /scripts/fe-member-leave.sh, container: fe
• Rendered CmpV: both 3.2.2 and 3.3.0 bind memberLeave to the exact FE image
• Fresh read-only design-contract review of the final two-blocker delta: APPROVED blocker0

An earlier whole-repository ShellSpec run produced 1008 examples, with 9 unrelated baseline failures in FalkorDB dependency and MariaDB render paths, plus 12 skips and 9 pending. This PR claims only the focused StarRocks-CE result above.

Production image gate

• starrocks/fe-ubuntu:3.2.2 digest sha256:6446acb1a16ce103476b17c3844e9f7e12cd09ac188cfe8ff01aad56ca87e612
• starrocks/fe-ubuntu:3.3.0 digest sha256:ef86f9cdf06bf71b01baedbd16bb72b662b7abe475f447ad4f4025e77ab2c40a
• Both contain /bin/bash, mysql, Java, GNU timeout, and exactly one readable StarRocks BDB JE jar.
• The mounted script was invoked through /bin/bash in both images and returned a classified rc1 on an unavailable endpoint without exposing the password.

Live acceptance plan

After product and lifecycle review approve this exact head, the test owner will package the exact commit and return: chart package SHA256, rendered memberLeave CmpD/CmpV structure, installed chart/source identity, live ComponentDefinition/ComponentVersion images, and fresh scale-in evidence. Existing frozen and fresh scenes remain untouched until that gate.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 351 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (4180185) to head (87fda89).

Files with missing lines	Patch %	Lines
...arrocks-ce/scripts-ut-spec/fe_member_leave_spec.sh	0.00%	237 Missing ⚠️
...e/scripts-ut-spec/fe_member_leave_contract_spec.sh	0.00%	114 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##            main   #3209    +/-   ##
======================================
  Coverage   0.00%   0.00%            
======================================
  Files        147     149     +2     
  Lines      23282   23633   +351     
======================================
- Misses     23282   23633   +351     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[leon-ape]

After DROP FOLLOWER succeeds against LEADER_HOST, this verification still queries the load-balanced FE_DISCOVERY_SERVICE_NAME. That Service continues to include the leaving Pod until memberLeave succeeds, so the query can reach the dropped FE and read its stale local membership view, falsely returning post-drop-membership-not-converged and repeatedly retrying the action.