fix(mariadb): make Galera preStop ordered and DNS-safe

[weicao]

Problems observed

This PR closes two evidence-backed failures in the same Galera native preStop path.

1. In the r21 frozen scene, Stop and Start OpsRequests both reported Succeed, but the data plane stayed failed. Every PVC recorded mysqladmin_shutdown_failed; pod-0 and pod-1 also exhausted the higher-ordinal wait. The hook had consumed the Pod grace budget with SQL and mysqladmin shutdown before kubelet could signal mariadbd PID 1.
2. After the first fix, the fresh r22 scene recovered to Primary/size3/all-Synced/rows9, but the hook contract still failed. pod-1 was deleted at 11:19:13Z and pod-2 at 11:19:20Z; pod-0 then waited the full 60 seconds, Kubernetes emitted FailedPreStopHook at 11:20:07Z, and pod-0's PVC persisted reason=order_wait_timeout with both deleted higher peers marked budget-exhausted. The hook treated their post-deletion DNS name absence as indefinite uncertainty.

The second result means r22 is runtime FAIL/N=1 for preStop validation, despite successful data-plane recovery.

Root cause

galera-prestop.sh is a Kubernetes native container preStop, not a kbagent lifecycle action. Kubernetes starts the Pod termination grace countdown before the hook and sends TERM to container PID 1 only after the hook returns.

The old code had two separate contract violations:

• it ran client-side SQL/mysqladmin shutdown inside the hook and swallowed failures, delaying kubelet's real TERM;
• its bounded TCP probe only recognized Connection refused as stopped. Once a deleted higher Pod disappeared from headless-Service DNS, Name or service not known / No address associated could never close that peer, so the lower ordinal exhausted the 60-second wait.

Primary contracts:

• Kubernetes native hook ordering and shared grace budget: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
• MariaDB Galera orderly shutdown / last clean node safe-to-bootstrap contract: https://mariadb.com/docs/galera-cluster/high-availability/resetting-the-quorum-cluster-bootstrap

Behavior fix

• Keep preStop responsible only for bounded high-to-low ordering and publishing .galera-shutting-down for the watcher guard.
• Let kubelet send TERM to the exec-owned mariadbd PID 1; do not run SQL or mysqladmin in the hook.
• Keep Connection refused as an immediate closed observation.
• Classify only authoritative name absence (Name or service not known / No address associated) as absent, and require the same peer to be absent in two consecutive polls before confirming it stopped.
• Clear that peer's absence streak on any live or uncertain observation (open, transient DNS failure, timeout, generic unreachable, or unknown) and continue fail-closed.
• Track peers independently, remember confirmed peers, preserve the existing wall-clock deadline, and return non-zero with persisted degraded evidence on unresolved peers.
• No K8s API, RBAC, sidecar, shared-volume coordination, or new engine shutdown mechanism is added.
• Bump the chart to 1.2.0-alpha.34 because the ComponentDefinition runtime contract changed.

The Chinese design document was published in the MariaDB team thread before this second code change. It covers the problem, alternatives, core flow, compatibility, and split Dev/Test plan.

TDD and local source validation

First fix:

• Old parent focused RED: 22 examples / 6 failures.
• Patched focused GREEN: 22 / 0.

DNS/deleted-peer fix on exact 90434973494b6b8207b287e7ea5bbd2cc3292307:

• Production RED with tests only: 28 examples / 6 failures; failures covered authoritative absence classification, transient classification, two-poll confirmation, uncertain-state reset, and independent peer state.
• Current Bash 3.2 focused GREEN: 29 / 0.
• Related Galera/switchover/version specs on Bash 5.3: 250 / 0 / 6 documented pending.
• Full MariaDB ShellSpec on Bash 5.3: 777 / 0 / 9 documented pending.
• bash -n, ShellCheck error gate, and git diff --check: PASS.
• Helm dependency build, lint, default render, and package: PASS.
• Offline package proof only: mariadb-1.2.0-alpha.34.tgz SHA256 51af67d5c73bf32a8e2957f1faabc4b881d10a45a8a5d62f7b35d7e9a6e2dbac; default render SHA256 95890869b273f7d8eba1e29813028a426596f02b3c374971643ea4112246e447.

Current boundary

• Exact head: c509f506861d21e88ce16f4498ac08ae3fce7e46.
• CI: rerunning for this exact head; not yet terminal at the time of this update.
• Runtime: no post-c509f5068 runtime result. The r22 scene remains frozen read-only and is not evidence for the new source.
• Merge order: this PR is stacked on PR mariadb: preserve secondary fence marker while pending #3216 and must follow it.
• A fresh Test-owned immutable package and Galera Stop/Start run must show no FailedPreStopHook, no new persisted degraded record, and the exact Primary/size3/Synced/rows oracle before any runtime-fixed claim.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 160 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (e32233b) to head (c509f50).

Files with missing lines	Patch %	Lines
...ons/mariadb/scripts-ut-spec/galera_prestop_spec.sh	0.00%	150 Missing ⚠️
...adb/scripts-ut-spec/replication_switchover_spec.sh	0.00%	5 Missing ⚠️
...scripts-ut-spec/galera_memberleave_removed_spec.sh	0.00%	3 Missing ⚠️
...ripts-ut-spec/replication_user_convergence_spec.sh	0.00%	2 Missing ⚠️

Additional details and impacted files

@@                             Coverage Diff                              @@
##           helen/mariadb-r18-secondary-marker-stability   #3217   +/-   ##
============================================================================
  Coverage                                          0.00%   0.00%           
============================================================================
  Files                                               154     154           
  Lines                                             24331   24409   +78     
============================================================================
- Misses                                            24331   24409   +78     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.