Skip to content

Commit 642b62e

Browse files
committed
Update README.md
1 parent 6d5573d commit 642b62e

1 file changed

Lines changed: 61 additions & 0 deletions

File tree

crates/consistent-choose-k/README.md

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,67 @@ replication: it is the unique `k`-out-of-`n` growth process where each new node
147147
joins the active set with probability `k/(n+1)`, evicts at most one old node,
148148
and preserves a uniform active set after every growth step.
149149

150+
## Fast incremental hashers
151+
152+
For workloads that repeatedly grow or shrink `n` at a fixed `k`, this crate
153+
ships two stateful specializations of `ConsistentChooseKHasher` whose
154+
amortized per-step cost is `O(log k)` rather than `O(k log k)` for a fresh
155+
`new_with_k`. They share the same input contract and produce the same set
156+
as the stateless version after every step — they just avoid recomputing
157+
the whole ranking from scratch.
158+
159+
| Type | Optimized for | Per-step cost | Construction |
160+
|---------------------------------|---------------|---------------|--------------|
161+
| `ConsistentChooseKHasher` | one-shot | `O(k log k)` total | `new_with_k(builder, n, k)` |
162+
| `ConsistentChooseKFastHasher` | repeated `shrink_n` | `O(log k)` amortized | `new_with_k(builder, n, k)` |
163+
| `ConsistentChooseKFastGrowHasher` | repeated `grow_n` | `O(log k)` amortized | `new(builder, k)` starts at `n = k` |
164+
165+
### Why two specializations?
166+
167+
`grow_n` and `shrink_n` keep different invariants, so they want different
168+
in-memory layouts and cannot easily share a single representation:
169+
170+
* **`ConsistentChooseKFastHasher`** (shrink): stores samples sorted by
171+
value, with a per-position count of "how many already-selected
172+
smaller samples block this slot". `shrink_n` is then a logarithmic
173+
segment-tree descent to the displaced slot.
174+
* **`ConsistentChooseKFastGrowHasher`** (grow): stores samples in
175+
insertion order, with a per-sample "life" `= seq_id - position`.
176+
An entry whose `life ≤ 0` is the slot to evict on the next firing.
177+
`grow_n` is a heap pop, a logarithmic rightmost-non-positive query
178+
on the sample structure, and a constant-time append.
179+
180+
Both are backed by an implicit-tree `SampleTreap` that supports
181+
`O(log k)` per operation for: insert, remove, point queries,
182+
range life additions with lazy propagation, and rightmost-`life ≤ 0`
183+
search via subtree-min augmentation.
184+
185+
### Sequence iteration: bucket-batch heap
186+
187+
Internally each of the `k` consistent-hash sequences advances through
188+
disjoint **buckets** covering value ranges `[b, 2b)` for powers of two
189+
`b`. The fast-grow hasher keeps a single min-heap keyed by
190+
`(sample, packed_seq)` where `packed_seq = seq * 2 + owner_bit`. When
191+
a seq's next bucket is loaded, every sample it produces is pushed at
192+
once, and the largest one is tagged with the owner bit. Popping that
193+
tagged entry is the signal to load the seq's next bucket. This
194+
amortizes the per-pop cost of the hash sequence over an entire bucket
195+
and avoids reconstructing per-seq iterator state.
196+
197+
### When to use which
198+
199+
* **Reservoir-style growth** (e.g. ingesting a stream and maintaining
200+
a uniform top-`k`): use `ConsistentChooseKFastGrowHasher`. It is the
201+
realization of the `O(k)`-per-step `grow_n` foreshadowed in the
202+
reservoir-sampling section, and is competitive with Algorithm R
203+
while remaining deterministically reproducible from `key` alone.
204+
(Algorithm L's geometric-skip stays faster when *all* you need is
205+
the sample; the choose-k variant pays for being history-independent.)
206+
* **Shrinking cluster / load-shedding**: use
207+
`ConsistentChooseKFastHasher`.
208+
* **Single point lookup** (one `(n, k)` per key, no follow-up):
209+
stick with the stateless `ConsistentChooseKHasher`.
210+
150211
## N-Choose-K replication
151212

152213
We define the consistent `n-choose-k` replication as follows:

0 commit comments

Comments
 (0)