redis-failure-lab

A hands-on laboratory for understanding Redis persistence, replication, and failure semantics through controlled experiments and Go tooling.

Project Structure

cmd/redislab/       CLI to run experiments by name
resp/               Minimal RESP protocol client (encode/decode + connection client)
inspector/          Redis state inspectors (persistence, replication)
experiment/         Experiment framework, all experiments, timeline renderer
docker/             Redis configuration files for each experiment topology
docs/               Failure modes documentation, architecture diagrams
docker-compose.yml  All Redis containers (persistence, replication, sentinel)

Quick Start

# Run unit tests
make test

# Run unit tests with coverage
make test-coverage

# Start Docker containers and run integration tests
make test-integration

# Stop containers
make docker-down

Packages

resp - RESP Protocol Client

Minimal, zero-dependency Redis client implementing RESP2 encoding/decoding. Supports all five RESP types: Simple String, Error, Integer, Bulk String, and Array.

inspector - Redis State Inspectors

Parses INFO persistence and INFO replication output into structured Go types. Includes polling watchers for observing state transitions over time.

experiment - Experiment Framework

Structured lifecycle for Redis experiments (Setup -> Run -> Observe -> Teardown) with timeline rendering and result formatting.

Available experiments:

• rdb-snapshot - Write data, trigger BGSAVE, verify snapshot creation
• aof-fsync - Compare always vs everysec fsync modes
• crash-recovery - Crash Redis during writes, observe RDB vs AOF recovery
• repl-offsets - Inspect replication offsets and backlog propagation
• partial-resync - Disconnect replica briefly, verify partial resync (PSYNC)
• full-resync - Overflow replication backlog, force full resync
• data-divergence - Write conflicting data during split, observe master authority
• sentinel-failover - Kill master, observe Sentinel election and promotion
• sentinel-splitbrain - Demonstrate split-brain data loss risk

Docker Topology

Container	Port	Purpose
redis-rdb	6380	RDB-only persistence
redis-aof-everysec	6381	AOF with everysec fsync
redis-aof-always	6382	AOF with always fsync
redis-master	6383	Replication master
redis-replica	6384	Replication replica
redis-sentinel-master	6385	Sentinel master
redis-sentinel-replica1	6386	Sentinel replica 1
redis-sentinel-replica2	6387	Sentinel replica 2
sentinel1	26379	Sentinel instance 1
sentinel2	26380	Sentinel instance 2
sentinel3	26381	Sentinel instance 3

CLI Usage

# Build the CLI
go build -o redislab ./cmd/redislab/

# List all experiments
./redislab -list

# Run a specific experiment
./redislab rdb-snapshot

# Run multiple experiments
./redislab repl-offsets partial-resync full-resync

# Run all experiments
./redislab all

# Custom timeout
./redislab -timeout=180s sentinel-failover

Documentation

• Failure Modes & Invariants - Catalog of every failure mode with invariants
• Architecture & Timelines - System diagrams and experiment timelines