feat: add HA with standby nodes (#54)

* first version of HA with standby node implemented with reverse proxy for request forwarding

* feat: HA with standby nodes, e2e test suite, and documentation

- Implement active/standby HA with mTLS reverse proxy forwarding
- Standby nodes forward requests to leader, serve health endpoints locally
- Preserve original Host header through proxy for AWS SigV4 compatibility
- Add cluster listener with self-signed TLS certificate generation
- Add leader advertisement and cluster address discovery

- Add comprehensive e2e test suite (11 packages, ~80 tests)
  covering cluster health, forwarding, SigV4, credentials,
  rotation, namespaces, providers, auth, audit, seal, concurrency
- Add e2e infrastructure: 3-node cluster setup/teardown/reset scripts,
  Go test helpers, and Docker Compose for dependencies

- Add Makefile targets: test-e2e, test-e2e-setup, test-e2e-teardown, test-e2e-reset
- Remove unused test-integration target
- Update README with HA section and CONTRIBUTING with e2e guide

* docs: add CHANGELOG.md with release history

Add changelog covering v0.2.0, v0.1.1, and v0.1.0 releases
following Keep a Changelog conventions.

---------

Co-authored-by: Stephane NANGUE <snangue@MacBook-Pro-de-Stephane.local>

Author: stephnangue

.github/workflows/ci.yml

@@ -23,3 +23,26 @@ jobs:
 
       - name: Build
         run: go build -o warden .
+
+  e2e:
+    runs-on: ubuntu-latest
+    needs: [test]
+    timeout-minutes: 45
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Set up Go
+        uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+
+      - name: Start E2E cluster
+        run: bash e2e/setup.sh
+
+      - name: Run E2E tests
+        run: go test -tags e2e -v -count=1 -p 1 -timeout 45m ./e2e/cluster/ ./e2e/provider/ ./e2e/ha/ ./e2e/namespace/ ./e2e/forwarding/ ./e2e/credential/ ./e2e/rotation/ ./e2e/auth/ ./e2e/audit/ ./e2e/seal/ ./e2e/concurrency/
+
+      - name: Teardown E2E cluster
+        if: always()
+        run: bash e2e/teardown.sh

CHANGELOG.md

@@ -0,0 +1,54 @@
+# Changelog
+
+All notable changes to Warden are documented in this file.
+
+## [v0.2.0] — 2026-03-04
+
+### New Features
+
+- **High Availability with Standby Nodes** — Active/standby HA using PostgreSQL advisory locks for leader election. Standby nodes forward requests to the leader via mTLS reverse proxy. Automatic failover when the leader becomes unavailable, with sealed-node protection to prevent forwarding to unhealthy nodes. Health and status endpoints (`sys/health`, `sys/leader`, `sys/seal-status`, `sys/init`, `sys/ready`) are served locally by standby nodes without forwarding. (#54)
+- **OpenAI AI Provider** — Native OpenAI provider with transparent gateway mode. (#52)
+- **Mistral AI Provider** — Native Mistral AI provider with transparent gateway mode. (#50)
+- **Opt-in Request Body Parsing for Streaming Requests** — Streaming requests can now opt in to request body parsing for policy evaluation while preserving the original stream. (#49)
+- **E2E Test Suite** — Comprehensive end-to-end tests running against a 3-node HA cluster covering cluster health, HA failover, request forwarding, provider integration, credential management, rotation, namespaces, seal/unseal, authentication, audit logging, and concurrency.
+
+### Bug Fixes
+
+- **SigV4 Host Header Preservation** — Fixed AWS SigV4 signature verification failure when requests are forwarded through standby nodes. The reverse proxy no longer rewrites the `Host` header, preserving the original value needed for signature verification. (#54)
+- **Dependabot Unblocked** — Fixed broken OpenBao sub-module references that prevented Dependabot from running. (#35)
+
+### Infrastructure
+
+- **Go 1.26.0** — Upgraded from Go 1.25.1. (#48)
+- **CI Updates** — Bumped `actions/checkout` to v6, `actions/setup-go` to v6, `goreleaser/goreleaser-action` to v7. (#36, #37, #38)
+- **Dependency Updates** — Updated `github.com/cloudflare/circl`, `github.com/go-chi/chi`, and various Go module dependencies. (#41, #42, #44, #47)
+
+## [v0.1.1] — 2025-12-22
+
+### Bug Fix
+
+- **fix: handle custom dev root tokens in LookupToken** — `LookupToken` failed with `"failed to detect token type"` when using `--dev-root-token` with a custom value that lacks a standard prefix. Added the same dev-mode fallback that `ResolveToken` already had. (#33)
+
+## [v0.1.0] — 2025-12-21
+
+Initial release. See the [v0.1.0 release notes](https://github.com/stephnangue/warden/releases/tag/v0.1.0) for the full feature list.
+
+### Highlights
+
+- Identity-aware egress gateway for cloud and SaaS services
+- Providers: AWS, Azure, GCP, GitHub, GitLab, Vault/OpenBao
+- Transparent and explicit gateway modes
+- JWT authentication with JWKS validation
+- Capability-based policy enforcement
+- Request-level audit trail
+- IP-bound sessions
+- Two-stage credential rotation
+- Seal/unseal with envelope encryption
+- Namespace isolation
+- Storage backends: in-memory, file, PostgreSQL
+- Docker image published to `ghcr.io/stephnangue/warden`
+- Pre-built binaries for Linux, macOS, and Windows
+
+[v0.2.0]: https://github.com/stephnangue/warden/compare/v0.1.1...v0.2.0
+[v0.1.1]: https://github.com/stephnangue/warden/compare/v0.1.0...v0.1.1
+[v0.1.0]: https://github.com/stephnangue/warden/releases/tag/v0.1.0

CONTRIBUTING.md

@@ -51,16 +51,62 @@ make brd-fast
 
 ## Running Tests
 
+### Unit Tests
+
 ```bash
 # Unit tests with race detection and coverage
 make test-unit
-
-# Integration tests
-make test-integration
 ```
 
 Coverage output is generated in `coverage.out`.
 
+### End-to-End Tests
+
+E2E tests run against a live 3-node HA cluster with Vault, Hydra, and PostgreSQL.
+
+| Command | Description |
+|---------|-------------|
+| `make test-e2e` | Start the cluster, run all e2e tests, tear down |
+| `make test-e2e-setup` | Start the e2e cluster only |
+| `make test-e2e-teardown` | Stop the e2e cluster |
+| `make test-e2e-reset` | Reset and restart the e2e cluster |
+
+To run a specific test suite or single test against an already-running cluster:
+
+```bash
+make test-e2e-setup
+go test -tags e2e -v ./e2e/forwarding/
+go test -tags e2e -run TestSigV4ThroughStandbyForwarding ./e2e/forwarding/ -v
+make test-e2e-teardown
+```
+
+#### E2E Test Suites
+
+| Package | Focus |
+|---------|-------|
+| `e2e/cluster` | Split-brain detection |
+| `e2e/ha` | Leader election, step-down, failover, node rejoin |
+| `e2e/forwarding` | Standby-to-leader request forwarding, SigV4 preservation |
+| `e2e/provider` | Vault transparent/non-transparent gateway, JWT validation |
+| `e2e/credential` | Credential issuance, caching, TTL expiry, cross-namespace |
+| `e2e/rotation` | Credential source rotation, activation delay, failover persistence |
+| `e2e/namespace` | Namespace CRUD and isolation |
+| `e2e/seal` | Seal/unseal operations |
+| `e2e/auth` | Authentication flows |
+| `e2e/audit` | Audit logging |
+| `e2e/concurrency` | Concurrent request handling |
+
+#### Writing E2E Tests
+
+- Use the `//go:build e2e` build tag
+- Import helpers: `h "github.com/stephnangue/warden/e2e/helpers"`
+- Use `h.GetLeaderPort(t)` and `h.GetStandbyPort(t)` to discover cluster topology
+- Register cleanup via `t.Cleanup` **before** creating resources to avoid orphans on partial failure
+- Accept `409` (conflict) in setup to be idempotent across test reruns
+- Use `h.GetLeaderPort(t)` at cleanup time (not a captured port) to handle leader changes
+
+See [e2e/README.md](e2e/README.md) for full cluster architecture and configuration details.
+
 ## Development Workflow
 
 1. **Start dependencies**:

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help build build-fast brd brd-fast build-no-cache up up-logs down logs logs-tail restart clean clean-all shell test test-unit test-integration query status rebuild rebuild-quick watch cache-info edit-config deps-up deps-down deps-logs reset-warden-db reset-warden-db-force warden-db-logs warden-db-shell
+.PHONY: help build build-fast brd brd-fast build-no-cache up up-logs down logs logs-tail restart clean clean-all shell test test-unit test-e2e test-e2e-setup test-e2e-teardown test-e2e-reset query status rebuild rebuild-quick watch cache-info edit-config deps-up deps-down deps-logs reset-warden-db reset-warden-db-force warden-db-logs warden-db-shell
 
 # Enable BuildKit for faster builds
 export DOCKER_BUILDKIT=1
@@ -46,10 +46,15 @@ help:
 	@echo ""
 	@echo "Testing:"
 	@echo "  make test-unit       - Run Go unit tests"
-	@echo "  make test-integration - Run integration tests"
 	@echo "  make test            - Test proxy connection"
 	@echo "  make query           - Run sample query"
 	@echo ""
+	@echo "E2E Testing (3-node HA cluster):"
+	@echo "  make test-e2e-setup    - Start the e2e cluster"
+	@echo "  make test-e2e          - Run all e2e tests"
+	@echo "  make test-e2e-teardown - Stop the e2e cluster"
+	@echo "  make test-e2e-reset    - Reset and restart the e2e cluster"
+	@echo ""
 	@echo "Maintenance:"
 	@echo "  make clean           - Clean containers and volumes"
 	@echo "  make clean-all       - Deep clean (including cache)"
@@ -63,11 +68,31 @@ test-unit:
 	@go test -v -race -coverprofile=coverage.out ./...
 	@echo "✓ All tests passed"
 
-# Run integration tests (if you have a separate integration test suite)
-test-integration:
-	@echo "Running integration tests..."
-	@go test -v -tags=integration ./...
-	@echo "✓ Integration tests passed"
+# Start the e2e 3-node HA cluster
+test-e2e-setup:
+	@echo "Starting e2e cluster..."
+	@bash e2e/setup.sh
+	@echo "✓ E2E cluster ready"
+
+# Run all e2e tests (starts cluster, runs tests, tears down)
+test-e2e: test-e2e-setup
+	@echo "Running e2e tests..."
+	@go test -tags e2e -v ./e2e/... || (bash e2e/teardown.sh && exit 1)
+	@bash e2e/teardown.sh
+	@echo "✓ E2E tests passed"
+
+# Stop the e2e cluster
+test-e2e-teardown:
+	@echo "Tearing down e2e cluster..."
+	@bash e2e/teardown.sh
+	@echo "✓ E2E cluster stopped"
+
+# Reset and restart the e2e cluster
+test-e2e-reset:
+	@echo "Resetting e2e cluster..."
+	@bash e2e/reset.sh
+	@bash e2e/setup.sh
+	@echo "✓ E2E cluster reset and ready"
 
 # Normal build with cache (runs tests first)
 build: test-unit

README.md

@@ -202,6 +202,7 @@ Explicit mode is **required for AWS** (SigV4 request signing means Warden must h
 | Open source | Yes (MPL-2.0) | Gateway only (MIT) | No |
 | Credential rotation | Two-stage async (prepare → activate) | N/A — virtual keys only | Automated via SaaS |
 | Streaming support | Full HTTP streaming (SSE, chunked) | Yes | N/A |
+| High availability | Active/standby with automatic failover | SaaS-managed | SaaS-managed |
 | Identity model | Single JWT for all providers | RBAC + SSO (Enterprise) | Per-provider identity mapping |
 
 ## Architecture
@@ -214,6 +215,7 @@ Warden is a reverse proxy written in Go. Each provider is registered as a stream
 - **IP-bound sessions** — Sessions are tied to the caller's IP. A stolen session token is useless from a different machine.
 - **Two-stage credential rotation** — Rotation is split into PREPARE (mint new credentials while old ones remain valid) and ACTIVATE (commit new credentials, destroy old ones). For eventually-consistent providers like AWS and Azure, Warden defers activation by a configurable propagation delay, eliminating the polling loops that cloud SDKs typically require.
 - **Seal/unseal model** — Like Vault, Warden protects secrets at rest using envelope encryption. Supports dev mode (in-memory) and production mode with multiple seal types (Shamir, Transit, AWS KMS, GCP KMS, Azure Key Vault, OCI KMS, PKCS11, KMIP) and PostgreSQL storage.
+- **Active/standby HA** — Multiple Warden nodes share a storage backend and use lock-based leader election. One node is active; the rest are hot standbys that forward requests and automatically promote on leader failure. Sealed nodes are prevented from acquiring the leadership lock, eliminating cluster stalls.
 - **Namespace isolation** — Every credential source, policy, and mount point is scoped to a namespace with hard boundaries. Policies cannot leak across namespaces.
 
 ## Getting Started
@@ -281,6 +283,35 @@ Production mode requires a configuration file and external dependencies (Postgre
 ./warden server --config=./warden.hcl
 ```
 
+### High Availability
+
+Warden supports active/standby HA. Multiple nodes share the same storage backend and use PostgreSQL advisory locks for leader election. One node becomes the active leader; the rest are hot standbys that automatically promote on leader failure.
+
+**How it works:**
+
+- **Standby forwarding** — Standby nodes forward all write and read requests to the active leader via mTLS reverse proxy. Clients can send requests to any node; the response is the same regardless of which node receives it.
+- **Automatic failover** — If the leader fails, a standby acquires the lock and promotes itself. Standby nodes detect the leader change and redirect their forwarding proxy to the new leader.
+- **Sealed node protection** — Sealed nodes are prevented from acquiring the leadership lock, ensuring only fully operational nodes can become leader.
+
+**Configuration** — each node needs `api_addr` (its own API address, used by the leader to advertise itself), `cluster_addr` (its mTLS cluster address for inter-node communication), and a shared storage backend with `ha_enabled`:
+
+```hcl
+api_addr     = "http://10.0.1.1:8400"
+cluster_addr = "https://10.0.1.1:8401"
+
+storage "postgres" {
+  connection_url = "postgres://warden:password@db:5432/warden?sslmode=require"
+  table          = "warden_store"
+  ha_table       = "warden_ha_locks"
+  ha_enabled     = "true"
+}
+
+listener "tcp" {
+  address     = "0.0.0.0:8400"
+  tls_enabled = false
+}
+```
+
 ### Configuration
 
 Warden uses HCL configuration files. See `warden.local.hcl` for a full example covering storage backend, listener, providers, and auth methods.
@@ -299,7 +330,7 @@ Warden uses HCL configuration files. See `warden.local.hcl` for a full example c
 
 **Security** — Rate limiting per identity, mTLS to upstream providers, audit log tamper detection
 
-**Operations** — Active/standby HA, Helm chart, Docker Compose quick start, Terraform module
+**Operations** — Helm chart, Docker Compose quick start, Terraform module
 
 **Developer experience** — Web UI, Swagger/OpenAPI spec, MCP server for AI agent frameworks, SDKs (Go, Python, TypeScript)
 
@@ -314,6 +345,7 @@ make deps-up          # Start development dependencies
 make brd-fast         # Build and run (skip tests)
 make dev-watch        # Hot reload for development
 make test-unit        # Run unit tests with race detection
+make test-e2e         # Run e2e tests (3-node HA cluster)
 ```
 
 ## License

auth/method/jwt/backend.go

@@ -6,6 +6,7 @@ import (
 	"crypto/x509"
 	"fmt"
 	"net/http"
+	"strings"
 	"sync"
 	"time"
 
@@ -259,8 +260,11 @@ func verifyJWKSURLReachable(ctx context.Context, jwksURL string, caPEM string) e
 	return verifyURLReachable(ctx, jwksURL, caPEM)
 }
 
-// verifyOIDCDiscoveryURLReachable checks that the OIDC discovery URL is reachable
+// verifyOIDCDiscoveryURLReachable checks that the OIDC discovery endpoint is reachable.
+// The oidcDiscoveryURL is the issuer URL (e.g., http://localhost:4444); this function
+// appends /.well-known/openid-configuration to match what the cap/jwt library does.
 func verifyOIDCDiscoveryURLReachable(ctx context.Context, oidcDiscoveryURL string, caPEM string) error {
-	return verifyURLReachable(ctx, oidcDiscoveryURL, caPEM)
+	wellKnown := strings.TrimSuffix(oidcDiscoveryURL, "/") + "/.well-known/openid-configuration"
+	return verifyURLReachable(ctx, wellKnown, caPEM)
 }