Proposal: Automated Release Qualification for Oxia

[mattisonchao]

Overview

This proposal describes how the okk verification system automates release qualification for Oxia. When an RC tag is pushed, the agent runs a full verification pipeline and produces a verdict — no human involvement during testing.

Related: Continuous Verification Pipeline

RC Flow

                    Developer pushes tag
                      v0.5.0-rc1
                          │
                          ▼
┌──────────────────────────────────────────┐
│  GitHub webhook: new tag event           │
└────────────────────┬─────────────────────┘
                     │
                     ▼
┌──────────────────────────────────────────┐
│  okk-agent detects RC tag               │
│  ├─ Parses version from tag             │
│  ├─ Loads verification plan             │
│  ├─ Creates tracking issue:             │
│  │   "RC Verification: oxia v0.5.0-rc1" │
│  └─ Starts pipeline                     │
└────────────────────┬─────────────────────┘
                     │
                     ▼
┌──────────────────────────────────────────┐
│  Verification Pipeline (days)            │
│  ├─ Phase 1: Baseline                    │
│  ├─ Phase 2: Chaos                       │
│  ├─ Phase 3: Scale                       │
│  ├─ Phase 4: Upgrade / Downgrade         │
│  ├─ Phase 5: Long soak                   │
│  │                                       │
│  │  Agent posts progress on tracking     │
│  │  issue throughout all phases          │
│  │                                       │
│  ├─ ✓ Invariants checked at every step   │
│  └─ If safety violation → abort early    │
└────────────────────┬─────────────────────┘
                     │
                     ▼
┌──────────────────────────────────────────┐
│  Verdict                                 │
│  ├─ All invariants held → APPROVED       │
│  │   → Agent creates GitHub release draft│
│  │   → Agent comments: "RC approved"     │
│  │                                       │
│  ├─ Safety violation → BLOCKED           │
│  │   → Agent comments: what failed       │
│  │   → Human investigates and fixes      │
│  │   → Push new RC tag to retry          │
│  └─ Close tracking issue with verdict    │
└──────────────────────────────────────────┘
                     │
                     ▼
┌──────────────────────────────────────────┐
│  Human reviews verdict                   │
│  └─ Publishes release (one click)        │
└──────────────────────────────────────────┘

Tag Convention

The agent reacts to specific tag patterns:

Tag pattern	Agent action
v*.*.*-rc*	Start full RC verification pipeline
v*.*.* (no rc)	Informational — verify upgrade from previous release

RC Tracking Issue

When an RC verification starts, the agent creates a dedicated tracking issue:

Title: 🤖 RC Verification: oxia v0.5.0-rc1
Labels: [rc-verification, automated]

All progress is posted as comments on this issue:

🤖 RC verification started for v0.5.0-rc1
Pipeline: baseline → chaos → scale → upgrade/downgrade → soak
Estimated duration: 7d

---

🤖 Phase 1 complete: Baseline
• 3 nodes, 4 testcases (JVM + Go)
• p99=4.1ms, ops=200/s, errors=0
• All invariants: ✓

---

🤖 Phase 2 complete: Chaos
• 12 events: 4x pod-kill, 4x network-partition, 4x clock-skew
• All recovered, max recovery time: 28s
• All invariants: ✓

---

🤖 Phase 3 complete: Scale
• 3→5→7→5→3
• Rebalance time: avg 2.1m
• All invariants: ✓

---

🤖 Phase 4 complete: Upgrade/Downgrade
• v0.4.2→v0.5.0: zero downtime ✓
• v0.5.0→v0.4.2: zero downtime ✓
• v0.4.2→v0.5.0: data consistent ✓
• All invariants: ✓

---

🤖 Phase 5 complete: Soak (5d)
• 2.8M total operations
• 23 chaos events, all recovered
• Memory stable, no goroutine leaks
• All invariants: ✓

---

🤖 VERDICT: APPROVED ✓

All invariants held throughout 7d verification.
Total operations: 3,241,000
Workers tested: JVM, Go

Release draft created: [link]

Failure Scenario

If an invariant is violated, the agent reports immediately and aborts:

🤖 INVARIANT VIOLATION: data-consistency

Phase 2 (Chaos), after network-partition:
• basic-jvm: AssertionFailure — key "/test/key-4521" expected value
  "abc123" but got "xyz789"
• 3 assertion failures in 30 seconds
• Pipeline ABORTED

Logs: [pod logs snippet]
Metrics: p99 spiked to 2.3s during partition

🤖 VERDICT: BLOCKED
Fix required before release.

The developer fixes the issue, pushes v0.5.0-rc2, and the agent starts a fresh verification.

Retry and Resume

Scenario	Agent behavior
New RC tag (-rc2) pushed	Start fresh verification from Phase 1
Agent pod restarts mid-verification	Resume from last completed phase (state persisted)
@okk-agent skip-to-soak	Skip remaining phases, jump to long soak
@okk-agent abort	Stop verification, mark as ABORTED

GitHub Release Draft

When the verdict is APPROVED, the agent creates a release draft:

Tag: v0.5.0
Title: Oxia v0.5.0
Body:
  ## Verification
  RC verification completed successfully.
  • Duration: 7d
  • Operations: 3.2M
  • Chaos events: 52, all recovered
  • Scale cycles: 3→5→7→5→3
  • Upgrade/downgrade: v0.4↔v0.5 verified
  • Workers: JVM, Go
  • All invariants held ✓

Draft: true (human must publish)

The human reviews the draft, adds release notes, and publishes. The agent never publishes a release — that's always a human decision.

CI/CD Integration

The RC verification can also be triggered from GitHub Actions:

# .github/workflows/rc-verification.yaml
on:
  push:
    tags: ['v*.*.*-rc*']

jobs:
  notify-agent:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger verification
        run: |
          # Agent picks up the tag event via webhook
          # This job just ensures the webhook fires
          echo "RC tag pushed: ${{ github.ref_name }}"
          echo "Agent will start verification automatically"

No CI job runs the actual verification — the agent does. CI just ensures the event is visible.

Implementation

1. Tag detection — agent watches for v*-rc* tags via GitHub webhook/polling
2. Pipeline state — persist phase completion in ConfigMap or agent state, so restarts can resume
3. Tracking issue — one issue per RC, all progress as comments
4. Release draft — use GitHub API to create draft release on approval
5. Abort handling — clean up testcases and chaos experiments on abort