feat(core): ShaMap Merkle tree & ledger hashing by e-desouza · Pull Request #151 · XRPLF/xrpl-rust

e-desouza · 2026-03-28T13:41:46Z

Summary

ShaMap radix-16 Merkle trie and ledger hashing for the XRP Ledger, per #150.

The tree uses HAMT bitmap-indexed inner nodes (u16 bitmap + compact Vec) so sparse nodes don't waste memory. Hash caching uses a dirty flag — mutations only invalidate the root-to-leaf path. Inner node hashing writes into a single 516-byte stack buffer and calls sha512half() once.

Also includes:

Merkle inclusion proofs (extract_proof() / verify_proof()) for SPV-style verification
Ledger hashing (ledger_hash(), transaction_tree_hash(), account_state_hash()) matching xrpl.js
no_std compatible (alloc only)
68 unit tests including ported xrpl.js test vectors (8 incremental ShaMap hashes + ledger header hash)
4 cargo-fuzz targets and 6 criterion benchmark groups

Closes #150

Test plan

cargo test --release -- shamap — 68 tests
cargo test --release --no-default-features --features embassy-rt,core,utils,wallet,models,helpers,websocket,json-rpc -- shamap — no_std variant
cargo clippy --features std,tokio-rt,embassy-rt,actix-rt,futures-rt,smol-rt,core,wallet,models,utils,helpers,json-rpc,websocket,cli -- -D warnings
cargo +nightly fuzz run shamap_order_independence -- -max_total_time=30 — zero crashes
cargo bench -- shamap

Port the xrpl.js ShaMap full-tree approach to Rust. The ShaMap is the core data structure used in the XRP Ledger for transaction trees and account state trees, producing the root hashes committed in ledger headers. Implements: - ShaMap tree with Box-allocated inner/leaf nodes (16-branch radix trie) - SHA-512Half streaming hasher (no_std compatible via sha2 crate) - HashPrefix constants for domain separation in all hash contexts - Ledger header hash computation - Transaction and account state tree hash helpers - Inclusion proof extraction and verification (ShaMapProof) - 42 unit tests covering correctness, order independence, proofs, and fuzz-style tampering detection - Criterion benchmarks for 100/1000/10000 item trees All code is no_std compatible (uses alloc for Vec/Box).

Fix trailing whitespace in nftoken_mint.rs and apply rustfmt across transaction models, CLI module, and test mock that had import ordering and line length issues.

…ctors Optimize the ShaMap with academic state-of-the-art techniques: - HAMT bitmap-indexed inner nodes (Bagwell 2001): u16 bitmap + compact Vec instead of fixed 16-slot arrays. ~70-80% memory reduction for sparse nodes. O(1) branch lookup via popcount. - Hash caching with dirty-flag propagation: Cell<Option<Hash256>> on every node. Mutations invalidate only the root-to-leaf path — O(log16 n). Repeated hash() calls are O(1). - Single-buffer inner hash: 516-byte stack buffer with zero-init so empty branches contribute ZERO_256 for free. One sha512half() call. - One-shot sha512half: Sha512::digest() instead of streaming new/update/finalize. - Merkle inclusion proofs with extract_proof() and verify_proof(). - add_item_no_index() for TRANSACTION_NO_METADATA leaf type. - Ported xrpl.js test vectors: 8 incremental ShaMap hashes + ledger header hash from ripple-binary-codec. - Extended benchmarks: 6 groups (build, cached hash, incremental add, proof extraction, proof verify, contains) at 100/1K/10K items. Closes #150

Four coverage-guided fuzz targets using libFuzzer + arbitrary: - shamap_order_independence: root hash is insertion-order independent - shamap_add_remove_roundtrip: add+remove extras restores original hash - shamap_proof_integrity: valid proofs verify, tampered proofs fail, missing items have no proof - shamap_cache_consistency: incremental tree hash equals fresh-build hash after random add/remove sequences Run with: cargo +nightly fuzz run <target> -- -max_total_time=30

e-desouza

Two correctness issues flagged inline: the fuzz cache-consistency check fires only after all ops (not per-mutation as documented), and add_item panics with an OOB index access on duplicate keys.

Review by Claude Opus 4.6 · Prompt: V12

fuzz/fuzz_targets/shamap_cache_consistency.rs

src/core/shamap/tree.rs

…ncy check add_item now detects duplicate keys and replaces the existing leaf instead of recursing infinitely until an OOB panic on nibble(). Move the fuzz cache-consistency assertion inside the per-op loop so transient hash divergences that self-correct mid-sequence are caught.

pdp2121 · 2026-04-08T15:15:35Z

/ai-review

xrplf-ai-reviewer

Three correctness issues and one API design gap flagged — see inline comments.

Review by Claude Opus 4.6 · Prompt: V13

xrplf-ai-reviewer · 2026-04-08T15:23:51Z

src/core/shamap/tree.rs

+        let existing = self.children.remove(idx);
+        self.bitmap &= !(1u16 << slot);
+
+        let mut new_inner = ShaMapInner::new(self.depth + 1);


u8 depth overflows silently in release builds — use checked_add or assert max depth (63 for 32-byte keys):

Suggested change

let mut new_inner = ShaMapInner::new(self.depth + 1);

let mut new_inner = ShaMapInner::new(self.depth.checked_add(1).expect("ShaMap depth overflow"));

xrplf-ai-reviewer · 2026-04-08T15:23:51Z

src/core/shamap/tree.rs

+        }
+        new_inner.add_item(leaf);
+
+        let new_idx = compact_index(self.bitmap, slot);


add_item silently overwrites duplicate keys with no return value — callers building canonical trees can't detect collisions. Consider returning bool (inserted vs. replaced) or an error on collision.

xrplf-ai-reviewer · 2026-04-08T15:23:51Z

src/core/shamap/tree.rs

+/// A level in a ShaMap inclusion proof.
+pub struct ProofLevel {
+    /// Which branch (0..15) was taken at this level.
+    pub nibble: u8,


nibble is a public u8 with no range constraint — if ProofLevel is deserialized from untrusted input with nibble > 15, verify_proof silently computes a wrong hash and always returns false. Add a validation guard in verify_proof: if level.nibble > 15 { return false; }

xrplf-ai-reviewer · 2026-04-08T15:23:51Z

benches/benchmarks.rs

+                map.add_item(*extra_idx, prefix, extra_data.clone());
+                let h = black_box(map.hash());
+                // Remove to reset for next iteration
+                map.remove_item(extra_idx);


Unchecked remove_item return — failed reset silently inflates the tree across iterations, skewing timing:

Suggested change

map.remove_item(extra_idx);

assert!(map.remove_item(extra_idx), "failed to remove extra item in benchmark");

e-desouza requested a review from LimpidCrypto as a code owner March 28, 2026 13:41

e-desouza requested review from Patel-Raj11, kuan121 and pdp2121 and removed request for LimpidCrypto March 28, 2026 13:42

e-desouza added 4 commits March 28, 2026 09:45

style: apply cargo fmt to pre-existing formatting issues

5bdb8bb

Fix trailing whitespace in nftoken_mint.rs and apply rustfmt across transaction models, CLI module, and test mock that had import ordering and line length issues.

e-desouza force-pushed the feat/shamap-tree branch from b09266c to 7a73c6a Compare March 28, 2026 13:45

e-desouza added 2 commits March 28, 2026 09:47

Merge branch 'main' into feat/shamap-tree

68824a6

Merge branch 'main' into feat/shamap-tree

d268dc0

e-desouza commented Apr 3, 2026

View reviewed changes

fuzz/fuzz_targets/shamap_cache_consistency.rs Outdated Show resolved Hide resolved

src/core/shamap/tree.rs Show resolved Hide resolved

e-desouza added 2 commits April 3, 2026 04:13

Merge remote-tracking branch 'origin/main' into feat/shamap-tree

ea904bb

xrplf-ai-reviewer bot reviewed Apr 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): ShaMap Merkle tree & ledger hashing#151

feat(core): ShaMap Merkle tree & ledger hashing#151
e-desouza wants to merge 8 commits intomainfrom
feat/shamap-tree

e-desouza commented Mar 28, 2026 •

edited

Loading

Uh oh!

e-desouza left a comment

Uh oh!

Uh oh!

Uh oh!

pdp2121 commented Apr 8, 2026

Uh oh!

xrplf-ai-reviewer bot left a comment

Uh oh!

xrplf-ai-reviewer bot Apr 8, 2026

Uh oh!

xrplf-ai-reviewer bot Apr 8, 2026

Uh oh!

xrplf-ai-reviewer bot Apr 8, 2026

Uh oh!

xrplf-ai-reviewer bot Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	let mut new_inner = ShaMapInner::new(self.depth + 1);
	let mut new_inner = ShaMapInner::new(self.depth.checked_add(1).expect("ShaMap depth overflow"));

	map.remove_item(extra_idx);
	assert!(map.remove_item(extra_idx), "failed to remove extra item in benchmark");

Conversation

e-desouza commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

e-desouza left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pdp2121 commented Apr 8, 2026

Uh oh!

xrplf-ai-reviewer bot left a comment

Choose a reason for hiding this comment

Uh oh!

xrplf-ai-reviewer bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

xrplf-ai-reviewer bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

xrplf-ai-reviewer bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

xrplf-ai-reviewer bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

e-desouza commented Mar 28, 2026 •

edited

Loading