Commit 955593c
committed
Merge #1670: Introduce
956d0a9 test(chain): Update test docs to stop referencing `get_chain_position` (志宇)
1508ae3 chore: Fix typos (志宇)
af92199 refactor(wallet): Reuse chain position instead of obtaining new one (Jiri Jakes)
caa0f13 docs(wallet): Explain `.take` usage (志宇)
1196405 refactor(chain): Reorganize `TxGraph::insert_anchor` logic for clarity (志宇)
4706315 chore(chain): Address `CanonicalIter` nitpicks (志宇)
68f7b77 test(chain): Add canonicalization test (志宇)
da0c43e refactor(chain)!: Rename `LastSeenIn` to `ObservedIn` (志宇)
d4102b4 perf(chain): add benchmarks for canonicalization logic (志宇)
e34024c feat(chain): Derive `Clone` on `IndexedTxGraph` (志宇)
e985445 docs: Add ADR for `O(n)` canonicalization algorithm (志宇)
4325e2c test(chain): Add transitive anchor tests (志宇)
8fbee12 feat(chain)!: rm `get_chain_position` and associated methods (志宇)
582d6b5 feat(chain)!: `O(n)` canonicalization algorithm (志宇)
f6192a6 feat(chain)!: Add `run_until_finished` methods (志宇)
0aa39f9 feat(chain)!: `TxGraph` contain anchors in one field (志宇)
Pull request description:
Fixes #1665
Replaces #1659
### Description
Previously, getting the canonical history of transactions/UTXOs required calling `TxGraph::get_chain_position` on each transaction. This was highly inefficient and resulted in an `O(n^2)` algorithm. The situation is especially problematic when we have many unconfirmed conflicts.
This PR introduces an `O(n)` algorithm to determine the canonical set of transactions in `TxGraph`. The algorithm's premise is as follows:
1. If transaction `A` is determined to be canonical, all of `A`'s ancestors must also be canonical.
2. If transaction `B` is determined to be NOT canonical, all of `B`'s descendants must also be NOT canonical.
3. If a transaction is anchored in the best chain, it is canonical.
4. If a transaction conflicts with a canonical transaction, it is NOT canonical.
5. A transaction with a higher last-seen has precedence.
6. Last-seen values are transitive. A transaction's collective last-seen value is the max of it's last-seen value and all of it's descendants.
We maintain two mutually-exclusive `txid` sets: `canoncial` and `not_canonical`.
Imagine a method `mark_canonical(A)` that is based on premise 1 and 2. This method will mark transaction `A` and all of it's ancestors as canonical. For each transaction that is marked canonical, we can iterate all of it's conflicts and mark those as `non_canonical`. If a transaction already exists in `canoncial` or `not_canonical`, we can break early, avoiding duplicate work.
This algorithm iterates transactions in 3 runs.
1. Iterate over all transactions with anchors in descending anchor-height order. For any transaction that has an anchor pointing to the best chain, we call `mark_canonical` on it. We iterate in descending-height order to reduce the number of anchors we need to check against the `ChainOracle` (premise 1). The purpose of this run is to populate `non_canonical` with all transactions that directly conflict with anchored transactions and populate `canonical` with all anchored transactions and ancestors of anchors transactions (transitive anchors).
2. Iterate over all transactions with last-seen values, in descending last-seen order. We can call `mark_canonical` on all of these that do not already exist in `canonical` or `not_canonical`.
3. Iterate over remaining transactions that contains anchors (but not in the best chain) and have no last-seen value. We treat these transactions in the same way as we do in run 2.
#### Benchmarks
Thank you to @ValuedMammal for working on this.
```sh
$ cargo bench -p bdk_chain --bench canonicalization
```
Benchmark results (this PR):
```
many_conflicting_unconfirmed::list_canonical_txs
time: [709.46 us 710.36 us 711.35 us]
many_conflicting_unconfirmed::filter_chain_txouts
time: [712.59 us 713.23 us 713.90 us]
many_conflicting_unconfirmed::filter_chain_unspents
time: [709.95 us 711.16 us 712.45 us]
many_chained_unconfirmed::list_canonical_txs
time: [2.2604 ms 2.2641 ms 2.2680 ms]
many_chained_unconfirmed::filter_chain_txouts
time: [3.5763 ms 3.5869 ms 3.5979 ms]
many_chained_unconfirmed::filter_chain_unspents
time: [3.5540 ms 3.5596 ms 3.5652 ms]
nested_conflicts_unconfirmed::list_canonical_txs
time: [660.06 us 661.75 us 663.60 us]
nested_conflicts_unconfirmed::filter_chain_txouts
time: [650.15 us 651.36 us 652.71 us]
nested_conflicts_unconfirmed::filter_chain_unspents
time: [658.37 us 661.54 us 664.81 us]
```
Benchmark results (master): https://github.com/evanlinjin/bdk/tree/fix/1665-master-bench
```
many_conflicting_unconfirmed::list_canonical_txs
time: [94.618 ms 94.966 ms 95.338 ms]
many_conflicting_unconfirmed::filter_chain_txouts
time: [159.31 ms 159.76 ms 160.22 ms]
many_conflicting_unconfirmed::filter_chain_unspents
time: [163.29 ms 163.61 ms 163.96 ms]
# I gave up running the rest of the benchmarks since they were taking too long.
```
### Notes to the reviewers
* ***PLEASE MERGE #1733 BEFORE THIS PR!*** We had to change the signature of `ChainPosition` to account for transitive anchors and unconfirmed transactions with no `last-seen` value.
* The canonicalization algorithm is contained in `/crates/chain/src/canonical_iter.rs`.
* Since the algorithm requires traversing transactions ordered by anchor height, and then last-seen values, we introduce two index fields in `TxGraph`; `txs_by_anchor` and `txs_by_last_seen`. Methods `insert_anchor` and `insert_seen_at` are changed to populate these index fields.
* An ADR is added: `docs/adr/0003_canonicalization_algorithm.md`. This is based on the work in #1592.
### Changelog notice
* Added: Introduce an `O(n)` canonicalization algorithm. This logic is contained in `/crates/chain/src/canonical_iter.rs`.
* Added: Indexing fields in `TxGraph`; `txs_by_anchor_height` and `txs_by_last_seen`. Pre-indexing allows us to construct the canonical history more efficiently.
* Removed: `TxGraph` methods: `try_get_chain_position` and `get_chain_position`. This is superseded by the new canonicalization algorithm.
### Checklists
#### All Submissions:
* [x] I've signed all my commits
* [x] I followed the [contribution guidelines](https://github.com/bitcoindevkit/bdk/blob/master/CONTRIBUTING.md)
* [x] I ran `cargo fmt` and `cargo clippy` before committing
#### New Features:
* [x] I've added tests for the new feature
* [x] I've added docs for the new feature
#### Bugfixes:
* [x] This pull request breaks the existing API
* [x] I've added tests to reproduce the issue which are now passing
* [x] I'm linking the issue being fixed by this PR
ACKs for top commit:
ValuedMammal:
ACK 956d0a9
nymius:
ACK 956d0a9
oleonardolima:
utACK 956d0a9
jirijakes:
ACK 956d0a9
Tree-SHA512: 44963224abf1aefb3510c59d0eb27e3a572cd16f46106fd92e8da2e6e12f0671dcc1cd5ffdc4cc80683bc9e89fa990eba044d9c64d9ce02abc29a08f4859b69eO(n) canonicalization algorithmFile tree
18 files changed
+1242
-606
lines changed- .github/workflows
- crates
- bitcoind_rpc/tests
- chain
- benches
- src
- tests
- common
- wallet
- src
- wallet
- tests
- docs/adr
- example-crates/example_cli/src
18 files changed
+1242
-606
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| 54 | + | |
| 55 | + | |
54 | 56 | | |
55 | 57 | | |
56 | 58 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
78 | 78 | | |
79 | 79 | | |
80 | 80 | | |
| 81 | + | |
| 82 | + | |
81 | 83 | | |
82 | 84 | | |
83 | 85 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
389 | 389 | | |
390 | 390 | | |
391 | 391 | | |
| 392 | + | |
392 | 393 | | |
393 | 394 | | |
394 | 395 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
0 commit comments