Feature: Active/Standby High Availability

[patsevanton]

Labels: enhancement, core/general

Problem

NORA currently runs as a single instance (documented in ARCHITECTURE.md). This means:

• Planned downtime during upgrades. Even with Kubernetes Recreate strategy, there is a window where the registry is unavailable. While client-side caches mitigate this for pulls, CI/CD pipelines actively publishing artifacts during a deployment window will fail.
• No fault tolerance. A hardware failure, OOM kill, or crash of the single instance causes a complete registry outage until the process restarts or a new pod is scheduled. For organizations using NORA as a critical path in their build infrastructure, this is a single point of failure.
• S3 storage already decouples state. NORA's architecture stores all durable state on S3-compatible object storage, and in-memory indexes are rebuilt on startup. This means the groundwork for running multiple instances against shared storage is largely already in place — the codebase is already defensively coded for multi-replica scenarios (per-version keys for npm packument, sparse Cargo index regeneration from storage, etc.).

Proposed Solution

Introduce active/standby support — a lightweight high-availability mode where one instance serves traffic (active) and one or more instances stand by, ready to take over if the active fails.

Core Design

1. Shared-nothing, shared-storage model. Both active and standby instances connect to the same S3-compatible backend. No state replication protocol is needed — S3 is the source of truth.

2. Leader election via distributed lock. Use S3 conditional writes (If-None-Match / PutIfAbsent) or a lightweight coordination mechanism (e.g., a lease object in S3 with TTL-based heartbeat) to elect the active instance. The standby instance continuously attempts to acquire the lock; if the active fails to renew its lease, the standby promotes itself.

3. Config-driven. A new top-level configuration section:

   [ha]
   enabled = true
   mode = "active-standby"            # future: "active-active"
   lease_ttl_secs = 15                # how long a lease is valid
   heartbeat_interval_secs = 5        # how often the active renews
   fence_token = true                 # use monotonic fencing tokens

4. Fencing tokens. The elected active holds a monotonic token. Any write operations to S3 include this token so that a stale (former) active cannot corrupt data after a split-brain scenario. This is especially important for operations that do read-modify-write on storage (e.g., npm packument merging).

5. Readiness probe integration. The standby instance exposes /healthz as 503 Standby and /readyz as 503 while in standby mode. Kubernetes readiness probes ensure only the active receives traffic. Upon promotion, the new active transitions its probes to 200.

6. Fast promotion. Since indexes are rebuilt from S3 on startup, a standby instance can periodically warm its in-memory indexes (read-only polling of storage) so that promotion is near-instant rather than requiring a full reindex.

Behavioral Guarantees

Scenario	Behavior
Active crashes	Standby acquires lease within lease_ttl_secs, starts serving traffic
Network partition (active isolated)	Active's lease expires; standby promotes. Former active sees lease loss and stops serving writes (fencing)
Planned rollout	New standby starts, acquires lease only after old active releases it on shutdown (SIGTERM handler releases lease)
Both instances healthy, no failover	Standby serves 503 on readiness; zero cost to active
S3 unavailable	Both instances degrade; active returns 502, standby cannot promote

Implementation Outline

1. ha/ module — lease.rs (S3-based lease acquisition/renewal), fence.rs (fencing token management), promotion.rs (index warming and readiness flip).
2. Startup change — Before binding the HTTP listener, attempt to acquire the lease. If acquired, proceed as active; otherwise, enter standby mode (serve admin/metrics only, poll for lease).
3. Graceful shutdown — On SIGTERM, release the S3 lease object so the standby can promote immediately rather than waiting for TTL expiry.
4. Metrics — nora_ha_role gauge (0=standby, 1=active), nora_ha_lease_renewal_total, nora_ha_lease_acquisition_duration_seconds, nora_ha_promotion_total.
5. Config validation — Warn if ha.enabled = true but storage backend is local filesystem (HA requires shared storage).

Alternatives Considered

1. Active/Active (load-balanced multi-replica). More complex — requires distributed locking on every write, not just leader election. Could be a future mode but adds latency to every publish operation. Active/standby is simpler and sufficient for most HA needs.

2. External coordination (etcd, Consul, ZooKeeper). Adds operational burden and external dependencies, contradicting NORA's "single binary, no external services" philosophy. S3-based leasing keeps the dependency surface unchanged.

3. Kubernetes-native HA (PodDisruptionBudget + Recreate). Already the documented approach. It reduces but does not eliminate downtime windows and provides no fault tolerance for unplanned failures.

4. Hot standby with WAL-based replication. Overly complex for an artifact registry where S3 already provides durable, consistent storage. Not justified by the read-heavy, write-light workload.

Related Registry

• Core/General

Additional Context

• The codebase is already partially prepared for this: per-registry defensive coding for multi-replica safety exists in proxy_coalesce.rs, registry/npm.rs, registry/cargo_registry.rs, and config/auth.rs.
• The backup.rs module provides disaster recovery but not live failover — active/standby complements it.
• S3 conditional write support (PutIfAbsent) is available in the object_store crate already used by NORA.
• This feature is critical for enterprise adoption where SLA requirements mandate <30s recovery time.

[devitway]

Thanks for the detailed write-up, Anton — this lands right on NORA's architecture. It's the documented upgrade path for a limitation we currently accept: process-local coordination (proxy_coalesce, the publish lock, circuit breaker, rate limiter) is safe only under a single writer, and active/standby preserves exactly that. Agreed on deferring active-active.

I'd suggest splitting the work into two phases:

• Phase 1 — conditional-write primitive in StorageBackend (put_if_absent / put_if_match, for both S3 and local). It's the prerequisite for a correct lease, and independently useful — it hardens the read-modify-write paths that can already race under any concurrency.
• Phase 2 — S3 lease + readiness flip (standby serves /readyz as 503, promotes on lease loss, releases on SIGTERM), built on Phase 1.
• Active-active deferred, as you proposed.

Marking this for the roadmap under that phased plan. If you're up for tackling Phase 1 as a self-contained PR, that'd be a great foundation — happy to review the trait design before you go deep.