Persist spks derived from `KeychainTxOutIndex` by evanlinjin · Pull Request #1963 · bitcoindevkit/bdk

evanlinjin · 2025-05-22T07:27:10Z

Replaces #1960
Fixes #1953
Fixes #1964

Description

Users with large wallet and/or complex descriptors may experience slow startup of KeychainTxOutIndex. This PR addresses this problem by providing the option to persist derived spks so that they no longer need to be re-derived on startup.

The IndexedTxGraph API has been changed for better ergonomics.

Compared to #1960, this is a more longterm solution that does not depend on multi-threading logic.

Changelog notice

Changed
  - `KeychainTxOutIndex::new` to take in an additional parameter `persist_spks` to control whether derived spks are cached and persisted across restarts. The default of `persist_spks` is false.
  - `KeychainTxOutIndex` methods (`lookahead_to_target, `next_unused_spk`, `reveal_next_spk`) now return changesets as they may derive spks to be persisted.
  - The `InsertDescriptorError` type now wraps descriptors in `Box` to reduce enum size.

Added
  - `spk_cache` field to `indexer::keychain_txout::ChangeSet` which persists derived spks.
  - `IndexedTxGraph::from_changeset` - allows constructing from `indexed_tx_graph::ChangeSet` and only indexing when ready.
  - `IndexedTxGraph::reindex` method.

Fixed
  - `KeychainTxOutIndex::reveal_to_target` so that it actually returns `None` if the `keychain` does not exist.

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo +nightly fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature

@ValuedMammal

Thanks to @ValuedMammal for the suggestions

* When merging changesets, assert that spk of a given descriptor id & derivation index does not get changed. * When reading spk from cache, check the spk by deriving it.

This incentivies constructing `KeychainTxOutIndex` from a changeset before inserting descriptors (to make use of the spk cache).

Also added staging changes to `ChangeSet::spk_cache`. This way, we can avoid returning `ChangeSet`s for `apply_changeset` and `insert_descriptor`. * `KeychainTxOutIndex::new` now takes in an additional `use_spk_cache` parameter. * Fixed `reveal_to_target` method to actually return `None` if the keychain does not exist.

* `new` is now intended to construct a fresh indexed-tx-graph * `from_changeset` is added for constructing indexed-tx-graph from a previously persisted changeset * added `reindex` for calling after indexer mutations that require it * reintroduce `Default` impl

crates/chain/src/rusqlite_impl.rs

Do not reference last revealed table, in case none are revealed. Correct SQL column name.

ValuedMammal

ACK 62767f0

notmandatory · 2025-05-27T01:12:55Z

~~Might have a problem with the sqlite store. I'm manually testing and looks like it's storing the same script for all indexes. I'll keep looking but wanted to give a headsup not to merge this yet.~~
Nevermind, it's working fine, my sqlite select just wan't showing the blog data.

notmandatory

ACK 62767f0

tested with bdk-cli with cache enabled and sqlite schema updated and new table is being populated.

ValuedMammal · 2025-05-30T17:34:01Z

crates/chain/src/indexed_tx_graph.rs

+    ///
+    /// let (graph, reindex_cs) =
+    ///     IndexedTxGraph::from_changeset(persisted_changeset, move |idx_cs| -> anyhow::Result<_> {
+    ///         // e.g. KeychainTxOutIndex needs descriptors that weren’t in its CS


// e.g. KeychainTxOutIndex needs descriptors that weren’t in its change set.

ValuedMammal · 2025-05-30T17:36:07Z

crates/chain/src/indexer/keychain_txout.rs


 impl<K> KeychainTxOutIndex<K> {
-    /// Construct a [`KeychainTxOutIndex`] with the given `lookahead`.
+    /// Construct a [`KeychainTxOutIndex`] with the given `lookahead` and `use_spk_cache` boolean.


/// Construct a [`KeychainTxOutIndex`] with the given `lookahead` and `persist_spks` boolean.

ValuedMammal · 2025-05-30T17:36:53Z

crates/chain/src/indexer/keychain_txout.rs

+    /// // Derive 20 future addresses per chain and persist + reload script pubkeys via ChangeSets:
+    /// let idx = KeychainTxOutIndex::<&'static str>::new(20, true);
+    ///
+    /// // Derive 10 future addresses per chain without persistence:


Per "keychain" ?

ValuedMammal · 2025-05-30T17:37:42Z

crates/chain/src/indexer/keychain_txout.rs

 impl<K: Clone + Ord + Debug> KeychainTxOutIndex<K> {
+    /// Construct `KeychainTxOutIndex<K>` from the given `changeset`.
+    ///
+    /// Shorthand for called [`new`] and then [`apply_changeset`].


Shorthand for "calling"...

ValuedMammal · 2025-05-30T17:39:14Z

crates/chain/src/indexer/keychain_txout.rs

+///
+/// It tracks:
+/// 1. `last_revealed`: the highest derivation index revealed per descriptor.
+/// 2. `spks`: the cache of derived script pubkeys to persist across runs.


/// 2. `spk_cache`: ...

This commit addresses some leftover comments from bitcoindevkit#1963.

`spk_cache_stage` was being used to hold newly derived SPKs before being transferred to the `ChangeSet`. As a result it became necessary to call `_empty_stage_into_changeset` whenever the internal state changed, requiring the developer to know which methods depended on the functionality, e.g. `scan_txout` and `reveal_next_spk`. Further, this altered the behavior of the `ChangeSet` in surprising ways. For example revealing 1 SPK might return additional scripts in the `ChangeSet` if they had previously been staged. The rationale in bitcoindevkit#1963 was to avoid returning a `ChangeSet` from `insert_descriptor`. We can do this by providing another method that returns a tuple of `(bool, ChangeSet)` so that descriptor insertion results in updating the `ChangeSet` directly. Any code using the original `insert_descriptor` can still access the `ChangeSet` using `initial_changeset` (which may include SPKs from other descriptors), or we can expose an accessor to get a reference to the `spk_cache`.

evanlinjin force-pushed the feature/spk_cache branch 4 times, most recently from 7afd8b6 to c279c99 Compare May 22, 2025 08:25

feat(chain)!: Persist spks derived from KeychainTxOutIndex

d299dae

evanlinjin force-pushed the feature/spk_cache branch from c279c99 to d299dae Compare May 22, 2025 08:41

evanlinjin requested a review from ValuedMammal May 22, 2025 10:50

evanlinjin assigned evanlinjin and ValuedMammal May 22, 2025

evanlinjin added the api A breaking API change label May 22, 2025

evanlinjin added this to BDK Chain May 22, 2025

evanlinjin moved this to In Progress in BDK Chain May 22, 2025

notmandatory added this to the Wallet 2.0.0 milestone May 22, 2025

fix(chain)!: API and logical issues in KeychainTxOutIndex

76875e7

Thanks to @ValuedMammal for the suggestions

evanlinjin force-pushed the feature/spk_cache branch from 90b6813 to 76875e7 Compare May 23, 2025 00:51

feat(chain): KeychainTxOutIndex: Debug build checks

d761265

* When merging changesets, assert that spk of a given descriptor id & derivation index does not get changed. * When reading spk from cache, check the spk by deriving it.

evanlinjin force-pushed the feature/spk_cache branch from fb838d9 to d761265 Compare May 23, 2025 01:36

feat(chain): Add KeychainTxOutIndex::from_changeset constructor

19e3b5d

This incentivies constructing `KeychainTxOutIndex` from a changeset before inserting descriptors (to make use of the spk cache).

evanlinjin force-pushed the feature/spk_cache branch from 9d97b29 to 4bc8cc0 Compare May 23, 2025 08:35

evanlinjin force-pushed the feature/spk_cache branch from 4bc8cc0 to bbbc054 Compare May 23, 2025 08:36

evanlinjin force-pushed the feature/spk_cache branch from bbbc054 to 3126cd2 Compare May 23, 2025 10:34

docs(chain): Adds docs for persisting spks in KeychainTxOutIndex

603f133

evanlinjin force-pushed the feature/spk_cache branch from 8fef7c7 to 603f133 Compare May 23, 2025 11:52

evanlinjin marked this pull request as ready for review May 23, 2025 12:12

ValuedMammal mentioned this pull request May 26, 2025

bench: Add reindex_tx_graph benchmark #1968

Merged

3 tasks

portlandhodl mentioned this pull request May 26, 2025

[ft] SMT Replenish with > 1000 lastrevealed index #1960

Closed

ValuedMammal requested changes May 26, 2025

View reviewed changes

crates/chain/src/rusqlite_impl.rs Outdated Show resolved Hide resolved

crates/chain/src/rusqlite_impl.rs Outdated Show resolved Hide resolved

fix(rusqlite_impl): Fix derived spks create table statement

62767f0

Do not reference last revealed table, in case none are revealed. Correct SQL column name.

ValuedMammal approved these changes May 26, 2025

View reviewed changes

ValuedMammal requested a review from LagginTimes May 26, 2025 15:50

notmandatory moved this from In Progress to Needs Review in BDK Chain May 26, 2025

notmandatory approved these changes May 27, 2025

View reviewed changes

ValuedMammal merged commit e217542 into bitcoindevkit:master May 27, 2025
19 checks passed

github-project-automation bot moved this from Needs Review to Done in BDK Chain May 27, 2025

ValuedMammal mentioned this pull request May 27, 2025

Release bdk_chain 0.23.0 #1969

Closed

39 tasks

ValuedMammal reviewed May 30, 2025

View reviewed changes

ValuedMammal added a commit to ValuedMammal/bdk that referenced this pull request May 30, 2025

docs: update docs after introducing spk cache

bb6d7c1

This commit addresses some leftover comments from bitcoindevkit#1963.

evanlinjin mentioned this pull request Jul 26, 2025

Generating SPK Index upon loading wallet is slow for large wallets with lots of addresses #1953

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Persist spks derived from `KeychainTxOutIndex`#1963

Persist spks derived from `KeychainTxOutIndex`#1963
ValuedMammal merged 8 commits intobitcoindevkit:masterfrom
evanlinjin:feature/spk_cache

evanlinjin commented May 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ValuedMammal left a comment

Uh oh!

notmandatory commented May 27, 2025 •

edited

Loading

Uh oh!

notmandatory left a comment

Uh oh!

Uh oh!

ValuedMammal May 30, 2025

Uh oh!

ValuedMammal May 30, 2025

Uh oh!

ValuedMammal May 30, 2025

Uh oh!

ValuedMammal May 30, 2025

Uh oh!

ValuedMammal May 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

evanlinjin commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changelog notice

Checklists

All Submissions:

New Features:

Uh oh!

Uh oh!

Uh oh!

ValuedMammal left a comment

Choose a reason for hiding this comment

Uh oh!

notmandatory commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

notmandatory left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ValuedMammal May 30, 2025

Choose a reason for hiding this comment

Uh oh!

ValuedMammal May 30, 2025

Choose a reason for hiding this comment

Uh oh!

ValuedMammal May 30, 2025

Choose a reason for hiding this comment

Uh oh!

ValuedMammal May 30, 2025

Choose a reason for hiding this comment

Uh oh!

ValuedMammal May 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

evanlinjin commented May 22, 2025 •

edited

Loading

notmandatory commented May 27, 2025 •

edited

Loading