Valthree is a strongly consistent, distributed, Valkey-compatible database.
Under the hood, Valkey nodes write all data directly to S3. Clusters offer strong serializable consistency and inherit S3's 99.999999999% durability. Applications connect to a Valthree cluster using any Valkey or Redis client library.
Important
We built Valthree to show off Antithesis, our platform for testing distributed systems. We've intentionally kept this project simple: it's complex enough to have bugs, but small enough to understand quickly. Please don't rely on Valthree in production!
For more on Valthree's design and tests, read on. If you'd rather see a mission-critical distributed database tested with Antithesis, head over to etcd.
Valthree clusters persist the whole key-value database as a single JSON file in object storage. Clusters preserve consistency with optimistic concurrency control:
- Servers handle writes with a read-modify-write cycle: Valthree reads the database file from object storage, modifies it, and writes it back with the
If-Matchheader. If another process has modified the file during the read-modify-write cycle, the file's ETag changes, the write fails, and the client receives an error. - Valthree serves reads directly from object storage (without any caching).
┌──────────┐ ┌─────────┐
┌──────────┐ │ │ │
──► 1) SET a b ──┐ ┌──────────┐ │ │ ┌─────── 2) ETag:123 {} ◄──────── │ S3 │
│ │ │ │ │ │ │ db.json │
◄──── 4) OK ◄────┘ │ valthree │ │─┘ └► 3) If-Match:123 {"a": "b"} ──► │ │
│ │─┘ │ │
└──────────┘ └─────────┘
This design is terrible for performance — all writes conflict with each other! — but it's simple enough to implement in just two files. Keeping the implementation small lets us focus on testing.
Valthree uses Antithesis to make sure that clusters remain consistent — even in the face of faulty networks, unreliable disks, unsynchronized clocks, and all the other indignities of production environments. Rather than maintaining a tightly-coupled, ever-growing pile of traditional tests, Antithesis lets us thoroughly test Valthree with just one black-box test.
Valthree's test relies on property-based testing. Instead of running a hard-coded series of commands and then asserting the exact state of the database, Valthree's test executes a randomly-generated workload. Periodically, the test verifies that the clients haven't observed any inconsistencies. When run in Antithesis's deterministic environment and driven by our autonomous exploration engine, this one test finds Valthree's deepest bugs, makes them perfectly reproducible, and even lets us interactively debug failures.
The best places to start browsing Valthree's code are the entrypoints for the server and the Antithesis test. On each commit, a Github Action builds them into a container (defined in Dockerfile.valthree) and pushes them to Antithesis's artifact registry. The same Github Action also pushes a Docker Compose file that stiches together MinIO, a three-node Valthree cluster, and the test workload. Antithesis spins up the whole system, thoroughly explores its behavior, and produces a report of any failures.
Valthree's code includes examples of:
- Using the Antithesis Github Action (docs)
- Defining custom properties with the Antithesis SDK (docs)
- Instrumenting a Go binary for coverage-assisted exploration (docs)
- Emulating an AWS service in Antithesis (docs)
- Maintaining a local test workload for quick iteration (though it doesn't have fault injection or intelligent exploration)
See the full Antithesis documentation for more information. If you'd prefer a live demo, let us know!
Offered under the MIT License.