|
1 | 1 | <div align="center"> |
2 | 2 | <img src="./figures/walrus1.png" |
3 | 3 | alt="walrus" |
4 | | - width="30%"> |
5 | | - <div>Walrus: A high performance storage engine in Rust</div> |
| 4 | + width="25%"> |
| 5 | + <div>Walrus: A Distributed Message Streaming Engine</div> |
6 | 6 |
|
7 | 7 | [](https://crates.io/crates/walrus-rust) |
8 | 8 | [](https://docs.rs/walrus-rust) |
|
11 | 11 |
|
12 | 12 | </div> |
13 | 13 |
|
14 | | -## Overview |
15 | 14 |
|
16 | 15 | Walrus is a distributed message streaming platform built on a high-performance log storage engine. It provides fault-tolerant streaming with automatic leadership rotation, segment-based partitioning, and Raft consensus for metadata coordination. |
17 | 16 |
|
@@ -76,10 +75,20 @@ make cluster-bootstrap |
76 | 75 | cargo run --bin walrus-cli -- --addr 127.0.0.1:9091 |
77 | 76 |
|
78 | 77 | # In the CLI: |
| 78 | + |
| 79 | +# create a topic named 'logs' |
79 | 80 | > REGISTER logs |
| 81 | + |
| 82 | +# produce a message to the topic |
80 | 83 | > PUT logs "hello world" |
| 84 | + |
| 85 | +# consume message from topic |
81 | 86 | > GET logs |
| 87 | + |
| 88 | +# get the segment states of the topic |
82 | 89 | > STATE logs |
| 90 | + |
| 91 | +# get cluster state |
83 | 92 | > METRICS |
84 | 93 | ``` |
85 | 94 |
|
@@ -183,6 +192,29 @@ make cluster-test-multi-topic # Multiple topics |
183 | 192 | - **Consensus overhead**: Metadata only (not data path) |
184 | 193 | - **Segment rollover**: ~1M entries default (~100MB depending on payload size) |
185 | 194 |
|
| 195 | +## Correctness |
| 196 | + |
| 197 | +Walrus includes a formal TLA+ specification of the distributed data plane that models segment-based sharding, lease-based write fencing, and cursor advancement across sealed segments. |
| 198 | + |
| 199 | +**Specification:** [distributed-walrus/spec/DistributedWalrus.tla](distributed-walrus/spec/DistributedWalrus.tla) |
| 200 | + |
| 201 | +### Verified Invariants |
| 202 | + |
| 203 | +- **Domain Consistency**: Topic metadata, WAL entries, and reader cursors stay synchronized |
| 204 | +- **Single Writer per Segment**: Only the designated leader can write to each segment |
| 205 | +- **No Writes Past Open Segment**: Closed segments remain immutable after rollover |
| 206 | +- **Sealed Counts Stable**: Entry counts for sealed segments match actual WAL contents |
| 207 | +- **Read Cursor Bounds**: Cursors never exceed segment boundaries or entry counts |
| 208 | +- **Sequential Write Order**: Entries within each segment maintain strict ordering |
| 209 | + |
| 210 | +### Liveness Properties |
| 211 | + |
| 212 | +- **Rollover Progress**: Segments exceeding the entry threshold eventually roll over |
| 213 | +- **Read Progress**: Available entries eventually get consumed by readers |
| 214 | + |
| 215 | +The specification abstracts Raft consensus as a single authoritative metadata source and models Walrus storage as per-segment entry sequences. Model checking with TLC verifies correctness under concurrent operations |
| 216 | + |
| 217 | + |
186 | 218 | ### Storage Engine Benchmarks |
187 | 219 |
|
188 | 220 | The underlying storage engine delivers exceptional performance: |
|
0 commit comments