feat: merge-train/barretenberg (#20677)

BEGIN_COMMIT_OVERRIDE
chore: update merkle tree audit scope (#20666)
feat!: translator revision (#20273)
END_COMMIT_OVERRIDE

Author: iakovenkos

barretenberg/cpp/scripts/audit/audit_scopes/merkle_tree_audit_scope.md

@@ -6,59 +6,53 @@ Commit hash: TBD (link)
 ## Files to Audit
 Note: Paths relative to `aztec-packages/barretenberg/cpp/src/barretenberg`
 
-### Core Merkle Tree Implementation
-1. `crypto/merkle_tree/merkle_tree.hpp`
-2. `crypto/merkle_tree/memory_tree.hpp`
-3. `crypto/merkle_tree/hash.hpp`
-4. `crypto/merkle_tree/hash_path.hpp`
-5. `crypto/merkle_tree/types.hpp`
-6. `crypto/merkle_tree/index.hpp`
-7. `crypto/merkle_tree/response.hpp`
-8. `crypto/merkle_tree/signal.hpp`
-
-### Storage Implementations
-9. `crypto/merkle_tree/memory_store.hpp`
-10. `crypto/merkle_tree/lmdb_store/lmdb_tree_store.cpp`
-11. `crypto/merkle_tree/lmdb_store/lmdb_tree_store.hpp`
-
-### Node Store
-12. `crypto/merkle_tree/node_store/array_store.hpp`
-13. `crypto/merkle_tree/node_store/cached_content_addressed_tree_store.hpp`
-14. `crypto/merkle_tree/node_store/content_addressed_cache.hpp`
-15. `crypto/merkle_tree/node_store/tree_meta.hpp`
+### Core Types and Utilities
+1. `crypto/merkle_tree/types.hpp`
+2. `crypto/merkle_tree/hash.hpp`
+3. `crypto/merkle_tree/hash_path.hpp`
+4. `crypto/merkle_tree/response.hpp`
+5. `crypto/merkle_tree/signal.hpp`
+
+### Storage Layer
+6. `crypto/merkle_tree/lmdb_store/lmdb_tree_store.hpp`
+7. `crypto/merkle_tree/lmdb_store/lmdb_tree_store.cpp`
+
+### Node Store (Cache + Metadata)
+8. `crypto/merkle_tree/node_store/cached_content_addressed_tree_store.hpp`
+9. `crypto/merkle_tree/node_store/content_addressed_cache.hpp`
+10. `crypto/merkle_tree/node_store/tree_meta.hpp`
 
 ### Append-Only Tree
-16. `crypto/merkle_tree/append_only_tree/content_addressed_append_only_tree.hpp`
+11. `crypto/merkle_tree/append_only_tree/content_addressed_append_only_tree.hpp`
 
 ### Indexed Tree
-17. `crypto/merkle_tree/indexed_tree/content_addressed_indexed_tree.hpp`
-18. `crypto/merkle_tree/indexed_tree/indexed_leaf.hpp`
-19. `crypto/merkle_tree/indexed_tree/fixtures.hpp`
-
-### Nullifier Tree
-20. `crypto/merkle_tree/nullifier_tree/nullifier_tree.cpp`
-21. `crypto/merkle_tree/nullifier_tree/nullifier_tree.hpp`
-22. `crypto/merkle_tree/nullifier_tree/nullifier_memory_tree.hpp`
-23. `crypto/merkle_tree/nullifier_tree/nullifier_leaf.hpp`
-
-### Test Fixtures
-24. `crypto/merkle_tree/fixtures.hpp`
-25. `crypto/merkle_tree/test_fixtures.hpp`
+12. `crypto/merkle_tree/indexed_tree/content_addressed_indexed_tree.hpp`
+13. `crypto/merkle_tree/indexed_tree/indexed_leaf.hpp`
 
 ## Summary of Module
 
-The `merkle_tree` module provides a comprehensive implementation of Merkle tree data structures for the Aztec protocol, including standard Merkle trees, append-only trees, indexed trees, and nullifier trees. The module supports multiple hash policies (Pedersen hash and Poseidon2) and provides both in-memory and persistent LMDB-backed storage options. The core implementation includes content-addressed storage with caching, allowing efficient tree operations and proof generation. The append-only tree implementation is optimized for sequential insertions, while the indexed tree maintains an ordered set of leaves with nullifier functionality. The nullifier tree is a specialized indexed tree used for tracking spent notes in the Aztec protocol. The module includes utilities for computing Merkle paths, tree roots, and managing tree metadata across multiple storage backends.
+The `merkle_tree` module implements the persistent Merkle tree infrastructure for the Aztec protocol's world state. It provides two tree variants: append-only trees (used for NOTE_HASH_TREE, L1_TO_L2_MESSAGE_TREE, and ARCHIVE) and indexed trees (used for NULLIFIER_TREE and PUBLIC_DATA_TREE). Both use Poseidon2 hashing.
+
+The storage architecture is two-layered: `ContentAddressedCachedTreeStore` wraps an in-memory cache (`ContentAddressedCache`) over a persistent LMDB backend (`LMDBTreeStore`). All tree operations are asynchronous, using thread pools and callback-based completion. The system supports block-level history, forking, checkpointing, and rollback for the world state managed by `world_state/`.
 
 ## Test Files
-1. `crypto/merkle_tree/merkle_tree.test.cpp`
-2. `crypto/merkle_tree/memory_tree.test.cpp`
-3. `crypto/merkle_tree/fixtures.test.cpp`
-4. `crypto/merkle_tree/lmdb_store/lmdb_tree_store.test.cpp`
-5. `crypto/merkle_tree/node_store/content_addressed_cache.test.cpp`
-6. `crypto/merkle_tree/append_only_tree/content_addressed_append_only_tree.test.cpp`
-7. `crypto/merkle_tree/indexed_tree/content_addressed_indexed_tree.test.cpp`
-8. `crypto/merkle_tree/nullifier_tree/nullifier_tree.test.cpp`
-9. `crypto/merkle_tree/nullifier_tree/nullifier_memory_tree.test.cpp`
+1. `crypto/merkle_tree/memory_tree.test.cpp`
+2. `crypto/merkle_tree/fixtures.test.cpp`
+3. `crypto/merkle_tree/lmdb_store/lmdb_tree_store.test.cpp`
+4. `crypto/merkle_tree/node_store/content_addressed_cache.test.cpp`
+5. `crypto/merkle_tree/append_only_tree/content_addressed_append_only_tree.test.cpp`
+6. `crypto/merkle_tree/indexed_tree/content_addressed_indexed_tree.test.cpp`
+7. `crypto/merkle_tree/nullifier_tree/nullifier_memory_tree.test.cpp`
+
+### Test Infrastructure
+Non-production code used for testing and benchmarking the production files above:
+
+- `crypto/merkle_tree/memory_tree.hpp` — Simple in-memory tree. Also used by vm2/simulation as a lightweight tree for AVM transaction execution.
+- `crypto/merkle_tree/fixtures.hpp` — Test utility functions (random directories, values, thread pools).
+- `crypto/merkle_tree/test_fixtures.hpp` — GTest assertion wrappers for tree state verification.
+- `crypto/merkle_tree/node_store/array_store.hpp` — Lightweight in-memory store substitute for LMDB in tests/benchmarks.
+- `crypto/merkle_tree/nullifier_tree/nullifier_memory_tree.hpp` — Reference implementation of an indexed nullifier tree, used for differential testing of `ContentAddressedIndexedTree`.
+- `crypto/merkle_tree/nullifier_tree/nullifier_leaf.hpp` — Leaf types used by `nullifier_memory_tree.hpp`.
 
 ## Security Mechanisms
 None identified.

barretenberg/cpp/scripts/test_chonk_standalone_vks_havent_changed.sh

@@ -13,7 +13,7 @@ cd ..
 # - Generate a hash for versioning: sha256sum bb-chonk-inputs.tar.gz
 # - Upload the compressed results: aws s3 cp bb-chonk-inputs.tar.gz s3://aztec-ci-artifacts/protocol/bb-chonk-inputs-[hash(0:8)].tar.gz
 # Note: In case of the "Test suite failed to run ... Unexpected token 'with' " error, need to run: docker pull aztecprotocol/build:3.0
-pinned_short_hash="600b85bd"
+pinned_short_hash="53ce2d4f"
 pinned_chonk_inputs_url="https://aztec-ci-artifacts.s3.us-east-2.amazonaws.com/protocol/bb-chonk-inputs-${pinned_short_hash}.tar.gz"
 
 script_path="$(cd "$(dirname "${BASH_SOURCE[0]}")/scripts" && pwd)/$(basename "${BASH_SOURCE[0]}")"

barretenberg/cpp/src/barretenberg/benchmark/CMakeLists.txt

@@ -7,7 +7,6 @@ add_subdirectory(chonk_bench)
 add_subdirectory(pippenger_bench)
 add_subdirectory(relations_bench)
 add_subdirectory(poseidon2_bench)
-add_subdirectory(merkle_tree_bench)
 add_subdirectory(indexed_tree_bench)
 add_subdirectory(append_only_tree_bench)
 add_subdirectory(ultra_bench)

barretenberg/cpp/src/barretenberg/benchmark/append_only_tree_bench/append_only_tree.bench.cpp

@@ -21,7 +21,6 @@ using namespace bb::crypto::merkle_tree;
 namespace {
 using StoreType = ContentAddressedCachedTreeStore<bb::fr>;
 
-using Pedersen = ContentAddressedAppendOnlyTree<StoreType, PedersenHashPolicy>;
 using Poseidon2 = ContentAddressedAppendOnlyTree<StoreType, Poseidon2HashPolicy>;
 
 const size_t TREE_DEPTH = 32;

barretenberg/cpp/src/barretenberg/benchmark/indexed_tree_bench/indexed_tree.bench.cpp

@@ -18,7 +18,6 @@ using namespace bb::crypto::merkle_tree;
 using StoreType = ContentAddressedCachedTreeStore<NullifierLeafValue>;
 
 using Poseidon2 = ContentAddressedIndexedTree<StoreType, Poseidon2HashPolicy>;
-using Pedersen = ContentAddressedIndexedTree<StoreType, PedersenHashPolicy>;
 
 const size_t TREE_DEPTH = 40;
 const size_t MAX_BATCH_SIZE = 64;

barretenberg/cpp/src/barretenberg/benchmark/merkle_tree_bench/CMakeLists.txt

@@ -6,16 +6,14 @@
 
 #pragma once
 #include "barretenberg/common/ref_vector.hpp"
-#include "barretenberg/common/throw_or_abort.hpp"
 #include "barretenberg/common/zip_view.hpp"
-#include "barretenberg/numeric/bitop/get_msb.hpp"
 #include <optional>
 
 namespace bb {
 
 /**
- * @brief Logic to support batching opening claims for unshifted, shifted and interleaved polynomials in Shplemini
- * @details Stores references to the commitments/evaluations of unshifted, shifted and interleaved polynomials to be
+ * @brief Logic to support batching opening claims for unshifted and shifted polynomials in Shplemini
+ * @details Stores references to the commitments/evaluations of unshifted and shifted polynomials to be
  * batch opened via Shplemini. Aggregates the commitments and batching scalars for each batch into the corresponding
  * containers for Shplemini. Computes the batched evaluation. Contains logic for computing the per-batch scalars
  * used to batch each set of claims (see details below).
@@ -35,26 +33,12 @@ template <typename Curve> struct ClaimBatcher_ {
         // scalar used for batching the claims, excluding the power of batching challenge \rho
         Fr scalar = 0;
     };
-    struct InterleavedBatch {
-        std::vector<RefVector<Commitment>> commitments_groups;
-        RefVector<Fr> evaluations;
-        std::vector<Fr> scalars_pos;
-        std::vector<Fr> scalars_neg;
-        Fr shplonk_denominator;
-    };
 
-    std::optional<Batch> unshifted;              // commitments and evaluations of unshifted polynomials
-    std::optional<Batch> shifted;                // commitments of to-be-shifted-by-1 polys, evals of their shifts
-    std::optional<InterleavedBatch> interleaved; // commitments to groups of polynomials to be combined by interleaving
-                                                 // and evaluations of the resulting interleaved polynomials
+    std::optional<Batch> unshifted; // commitments and evaluations of unshifted polynomials
+    std::optional<Batch> shifted;   // commitments of to-be-shifted-by-1 polys, evals of their shifts
 
     Batch get_unshifted() { return (unshifted) ? *unshifted : Batch{}; }
     Batch get_shifted() { return (shifted) ? *shifted : Batch{}; }
-    InterleavedBatch get_interleaved() { return (interleaved) ? *interleaved : InterleavedBatch{}; }
-    uint32_t get_groups_to_be_interleaved_size()
-    {
-        return (interleaved) ? static_cast<uint32_t>(interleaved->commitments_groups[0].size()) : 0;
-    }
 
     Fr get_unshifted_batch_scalar() const { return unshifted ? unshifted->scalar : Fr{ 0 }; }
 
@@ -97,52 +81,25 @@ template <typename Curve> struct ClaimBatcher_ {
             shifted->scalar =
                 r_challenge.invert() * (inverse_vanishing_eval_pos - nu_challenge * inverse_vanishing_eval_neg);
         }
-
-        if (interleaved) {
-            const size_t interleaving_denominator_index = 2 * numeric::get_msb(get_groups_to_be_interleaved_size());
-
-            if (get_groups_to_be_interleaved_size() % 2 != 0) {
-                throw_or_abort("Interleaved groups size must be even");
-            }
-
-            Fr r_shift_pos = Fr(1);
-            Fr r_shift_neg = Fr(1);
-            interleaved->shplonk_denominator = inverted_vanishing_evals[interleaving_denominator_index];
-            for (size_t i = 0; i < get_groups_to_be_interleaved_size(); i++) {
-                interleaved->scalars_pos.push_back(r_shift_pos);
-                interleaved->scalars_neg.push_back(r_shift_neg);
-                if (i < get_groups_to_be_interleaved_size() - 1) {
-                    // to avoid unnecessary multiplication gates in a circuit
-                    r_shift_pos *= r_challenge;
-                    r_shift_neg *= (-r_challenge);
-                }
-            }
-        }
     }
     /**
-     * @brief Append the commitments and scalars from each batch of claims to the Shplemini, vectors which subsequently
+     * @brief Append the commitments and scalars from each batch of claims to the Shplemini vectors which subsequently
      * will be inputs to the batch mul;
      * update the batched evaluation and the running batching challenge (power of rho) in place.
      *
      * @param commitments commitment inputs to the single Shplemini batch mul
      * @param scalars scalar inputs to the single Shplemini batch mul
      * @param batched_evaluation running batched evaluation of the committed multilinear polynomials
      * @param rho multivariate batching challenge \rho
-     * @param rho_power current power of \rho used in the batching scalar
-     * @param shplonk_batching_pos and @param shplonk_batching_neg consecutive powers of the Shplonk batching
-     * challenge ν for the interleaved contributions
      */
     void update_batch_mul_inputs_and_batched_evaluation(std::vector<Commitment>& commitments,
                                                         std::vector<Fr>& scalars,
                                                         Fr& batched_evaluation,
-                                                        const Fr& rho,
-                                                        Fr shplonk_batching_pos = { 0 },
-                                                        Fr shplonk_batching_neg = { 0 })
+                                                        const Fr& rho)
     {
         size_t num_powers = 0;
         num_powers += unshifted.has_value() ? unshifted->commitments.size() : 0;
         num_powers += shifted.has_value() ? shifted->commitments.size() : 0;
-        num_powers += interleaved.has_value() ? interleaved->evaluations.size() : 0;
 
         Fr rho_power = Fr(1);
         size_t power_idx = 0;
@@ -171,29 +128,6 @@ template <typename Curve> struct ClaimBatcher_ {
             // i-th shifted commitments will be multiplied by ρ^{num_unshifted + i} and r⁻¹ ⋅ (1/(z−r) − ν/(z+r))
             aggregate_claim_data_and_update_batched_evaluation(*shifted);
         }
-        if (interleaved) {
-            if (get_groups_to_be_interleaved_size() % 2 != 0) {
-                throw_or_abort("Interleaved groups size must be even");
-            }
-
-            size_t group_idx = 0;
-            for (size_t j = 0; j < interleaved->commitments_groups.size(); j++) {
-                for (size_t i = 0; i < get_groups_to_be_interleaved_size(); i++) {
-                    // The j-th commitment in group i is multiplied by ρ^{m+i} and ν^{d+1} \cdot r^j + ν^{d+2} ⋅(-r)^j
-                    //  where d is the log_circuit_size
-                    commitments.emplace_back(std::move(interleaved->commitments_groups[j][i]));
-                    scalars.emplace_back(-rho_power * interleaved->shplonk_denominator *
-                                         (shplonk_batching_pos * interleaved->scalars_pos[i] +
-                                          shplonk_batching_neg * interleaved->scalars_neg[i]));
-                }
-                batched_evaluation += interleaved->evaluations[group_idx] * rho_power;
-                power_idx++;
-                if (power_idx < num_powers) {
-                    rho_power *= rho;
-                }
-                group_idx++;
-            }
-        }
 
         BB_ASSERT_EQ(power_idx, num_powers);
     }