starknet_committer: add index layout leaves

[ArielElp]

No description provided.

[reviewable-StarkWare]

This change is 

[ArielElp]

• starknet_committer,starknet_patricia: move node_serde to facts db #11644
• starknet_committer: reorganize db modules #11640
• starknet_committer: add commit e2e tests #11331
• index layout forest creation tests #11286
• starknet_committer,starknet_patricia: layout-based original tree creation tests #11236
• starknet_committer: add original tree creation test templates #11715
• starknet_committer,starknet_patricia: layout-dependent db-key separator #11641
• starknet_committer,starknet_patricia: remove FilledNode impl #11637
• starknet_committer: db layout trait #11621
• starknet_committer,starknet_patricia: layout-dependent FilledTree serialization #10988
• starknet_committer,starknet_patricia: refactor key context #11592
• starknet_committer,starknet_patricia: add layout for base trait #11566
• starknet_committer,starknet_patricia: add serialization utilities to node layout #10987
• starknet_committer,starknet_patricia: move db_layout #10986
• starknet_committer: add index db #10685
• starknet_committer: abstract original tree creation #10684
• starknet_committer: enrich layout trait to abstract skeleton creation #11182
• starknet_committer: move trie creation logic to trie_traversal #11180
• starknet_committer: group generic layout logic #10852
• starknet_committer,starknet_patricia: remove generic config #10683
• starknet_committer: add index layout nodes #10682
• starknet_committer: add index layout subtree #10919
• starknet_committer: hash leaves of both layouts #10918
• starknet_committer: add index layout leaves #10681 👈 (View in Graphite)
• starknet_committer,starknet_patricia,starknet_patricia_storage: support contract address prefix #10858
• starknet_committer,starknet_patricia: add serde errors #10810
• starknet_committer,starknet_committer_and_os_cli,starknet_patricia: layout dependent fetch_nodes #10680
• starknet_committer,starknet_patricia: enrich subtree trait to support fetch nodes #10751
• starknet_committer,starknet_committer_and_os_cli,starknet_patricia: make FilledNode generic #10750
• starknet_committer,starknet_os,starknet_patricia: make node data generic #10679
• starknet_committer,starknet_patricia: add deserialization context #10704
• starknet_committer,starknet_patricia,starknet_patricia_storage: add context to prefix and serde #10678
• starknet_committer,starknet_patricia: add subtree trait #10694
• starknet_committer: reorganize facts db files #10677
• main-v0.14.1-committer

This stack of pull requests is managed by Graphite. Learn more about stacking.

[dorimedini-starkware]

@dorimedini-starkware reviewed 6 files and all commit messages, and made 8 comments.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @ArielElp).

---

crates/starknet_committer/src/db/index_db/leaves.rs line 41 at r8 (raw file):

        match self {
            TrieType::ContractsTrie => DbKeyPrefix::new(b"CONTRACTS_TREE_PREFIX".into()),
            TrieType::ClassesTrie => DbKeyPrefix::new(b"CLASSES_TREE_PREFIX".into()),

didn't we want these to be MAX_CONTRACT_ADDRESS + X?
also for metadata prefixes?

Code quote:

            TrieType::ContractsTrie => DbKeyPrefix::new(b"CONTRACTS_TREE_PREFIX".into()),
            TrieType::ClassesTrie => DbKeyPrefix::new(b"CLASSES_TREE_PREFIX".into()),

---

crates/starknet_committer/src/db/index_db/leaves.rs line 42 at r8 (raw file):

            TrieType::ContractsTrie => DbKeyPrefix::new(b"CONTRACTS_TREE_PREFIX".into()),
            TrieType::ClassesTrie => DbKeyPrefix::new(b"CLASSES_TREE_PREFIX".into()),
            TrieType::StorageTrie(contract_address) => {

Suggestion:

            Self::ContractsTrie => DbKeyPrefix::new(b"CONTRACTS_TREE_PREFIX".into()),
            Self::ClassesTrie => DbKeyPrefix::new(b"CLASSES_TREE_PREFIX".into()),
            Self::StorageTrie(contract_address) => {

---

crates/starknet_committer/src/db/index_db/leaves.rs line 117 at r8 (raw file):

        let nonce = deserialize_felt(&mut cursor, || {
            DeserializationError::FeltDeserialization(value.clone())
        })?;

please avoid cloning the value - you can implement some deserialize_felts(value: &DbValue, n: usize) -> Vec<Felt> to reuse the value, right?

Code quote:

        let mut cursor: &[u8] = &value.0;
        let class_hash = deserialize_felt(&mut cursor, || {
            DeserializationError::FeltDeserialization(value.clone())
        })?;
        let storage_root_hash = deserialize_felt(&mut cursor, || {
            DeserializationError::FeltDeserialization(value.clone())
        })?;
        let nonce = deserialize_felt(&mut cursor, || {
            DeserializationError::FeltDeserialization(value.clone())
        })?;

---

crates/starknet_committer/src/db/index_db/leaves.rs line 167 at r8 (raw file):

}

fn serialize_felts(felts: Vec<Felt>) -> Result<DbValue, SerializationError> {

must this be by value? slice is not enough?

Suggestion:

felts: &[Felt]

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 22 at r8 (raw file):

        storage_root_hash: HashOutput(Felt::from(2)),
        nonce: Nonce(Felt::from(3)),}
))]

non-blocking

Suggestion:

#[case::index_layout_contract_state(IndexLayoutContractState(ContractState {
    class_hash: ClassHash(Felt::from(1)),
    storage_root_hash: HashOutput(Felt::from(2)),
    nonce: Nonce(Felt::from(3)),
}))]

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 28 at r8 (raw file):

#[case::index_layout_starknet_storage_value(IndexLayoutStarknetStorageValue(
    StarknetStorageValue(Felt::from(1))
))]

also non-blocking

Suggestion:

#[case::index_layout_contract_state(IndexLayoutContractState(ContractState {
        class_hash: ClassHash(Felt::ONE),
        storage_root_hash: HashOutput(Felt::TWO),
        nonce: Nonce(Felt::THREE),}
))]
#[case::index_layout_compiled_class_hash(IndexLayoutCompiledClassHash(CompiledClassHash(
    Felt::ONE
)))]
#[case::index_layout_starknet_storage_value(IndexLayoutStarknetStorageValue(
    StarknetStorageValue(Felt::ONE)
))]

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 33 at r8 (raw file):

    let deserialized = L::deserialize(&serialized, &EmptyDeserializationContext).unwrap();
    assert_eq!(leaf, deserialized);
}

please add some regression-tests for serialized values, to make sure they are stable + nothing unexpected happens

#[rstest]
#[case(..)]
..
fn test_leaf_serialization_regression(#[case] leaf) {
    expect!["bla"].assert_debug_eq(leaf.serialize());
}

Code quote:

#[rstest]
#[case::index_layout_contract_state(IndexLayoutContractState(ContractState {
        class_hash: ClassHash(Felt::from(1)),
        storage_root_hash: HashOutput(Felt::from(2)),
        nonce: Nonce(Felt::from(3)),}
))]
#[case::index_layout_compiled_class_hash(IndexLayoutCompiledClassHash(CompiledClassHash(
    Felt::from(1)
)))]
#[case::index_layout_starknet_storage_value(IndexLayoutStarknetStorageValue(
    StarknetStorageValue(Felt::from(1))
))]
fn test_index_layout_leaf_serde<L: Leaf>(#[case] leaf: L) {
    let serialized = leaf.serialize().unwrap();
    let deserialized = L::deserialize(&serialized, &EmptyDeserializationContext).unwrap();
    assert_eq!(leaf, deserialized);
}

---

crates/starknet_committer/Cargo.toml line 32 at r8 (raw file):

tokio = { workspace = true, features = ["rt"] }
tracing.workspace = true
derive_more.workspace = true

alphabetize deps

Code quote:

derive_more.workspace = true

[dorimedini-starkware]

@dorimedini-starkware partially reviewed 6 files.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @ArielElp).

[ArielElp]

@ArielElp made 6 comments.
Reviewable status: 6 of 9 files reviewed, 6 unresolved discussions (waiting on @dorimedini-starkware and @yoavGrs).

---

crates/starknet_committer/Cargo.toml line 32 at r8 (raw file):

Previously, dorimedini-starkware wrote…

alphabetize deps

Done.

---

crates/starknet_committer/src/db/index_db/leaves.rs line 41 at r8 (raw file):

Previously, dorimedini-starkware wrote…

didn't we want these to be MAX_CONTRACT_ADDRESS + X?
also for metadata prefixes?

right, forgot, should be ok now

---

crates/starknet_committer/src/db/index_db/leaves.rs line 117 at r8 (raw file):

Previously, dorimedini-starkware wrote…

please avoid cloning the value - you can implement some deserialize_felts(value: &DbValue, n: usize) -> Vec<Felt> to reuse the value, right?

Note that cloning only happens in the error path, zero clones in the happy path. Since the error owns DbValue and I don't want to add a lifetime var there, I think we have to clone. I cleaned it a bit so the closure is defined once.

---

crates/starknet_committer/src/db/index_db/leaves.rs line 167 at r8 (raw file):

Previously, dorimedini-starkware wrote…

must this be by value? slice is not enough?

Yep, can be done with slices

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 33 at r8 (raw file):

Previously, dorimedini-starkware wrote…

please add some regression-tests for serialized values, to make sure they are stable + nothing unexpected happens

#[rstest]
#[case(..)]
..
fn test_leaf_serialization_regression(#[case] leaf) {
    expect!["bla"].assert_debug_eq(leaf.serialize());
}

Done. I used the same simple leaves, we could have fancier values (to see some compression and not just removal of leading zeros), but this would essentially be testing Felt serialization.

---

crates/starknet_committer/src/db/index_db/leaves.rs line 42 at r8 (raw file):

            TrieType::ContractsTrie => DbKeyPrefix::new(b"CONTRACTS_TREE_PREFIX".into()),
            TrieType::ClassesTrie => DbKeyPrefix::new(b"CLASSES_TREE_PREFIX".into()),
            TrieType::StorageTrie(contract_address) => {

Done.

[dorimedini-starkware]

@dorimedini-starkware reviewed 3 files and all commit messages, made 3 comments, and resolved 5 discussions.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @ArielElp).

---

crates/starknet_committer/src/db/index_db/leaves.rs line 117 at r8 (raw file):

Previously, ArielElp wrote…

Note that cloning only happens in the error path, zero clones in the happy path. Since the error owns DbValue and I don't want to add a lifetime var there, I think we have to clone. I cleaned it a bit so the closure is defined once.

ahhhh right, my bad

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 33 at r8 (raw file):

Previously, ArielElp wrote…

Done. I used the same simple leaves, we could have fancier values (to see some compression and not just removal of leading zeros), but this would essentially be testing Felt serialization.

1. please add some cases that indicate the byte order: i.e. larger values
2. how is it possible that the ContractState case serializes into just 3 bytes...? don't you need to measure the byte length of a value to know how to deserialize it? if the actual byte size of the three values are 10, 10 and 12, you will get 32 bytes, and try to deserialize as a binary node..?

---

crates/starknet_committer/src/db/index_db/leaves.rs line 39 at r9 (raw file):

/// Set to 2^251 + 2 to avoid collisions with contract addresses prefixes.
const CLASSES_TREE_PREFIX: &[u8] =
    b"0800000000000000000000000000000000000000000000000000000000000010";

1. reuse existing constant
2. if const doesn't work use LazyLock (see existing examples in our code)
3. not sure how you should be converting Felt to bytes, I just gave an example here; also you can consider just defining Felts in this context and converting to bytes only in the db_prefix method

Suggestion:

/// Set to 2^251 + 1 to avoid collisions with contract addresses prefixes.
const FIRST_AVAILABLE_PREFIX_FELT: Felt = Felt::from_hex_unchecked(PATRICIA_KEY_UPPER_BOUND) + Felt::ONE;

/// The db key prefix of nodes in the contracts trie.
const CONTRACTS_TREE_PREFIX: &[u8] = FIRST_AVAILABLE_PREFIX_FELT.to_bytes_be();

/// The db key prefix of nodes in the classes trie.
const CLASSES_TREE_PREFIX: &[u8] = (FIRST_AVAILABLE_PREFIX_FELT + Felt::ONE).to_bytes_be();

[ArielElp]

@ArielElp made 2 comments.
Reviewable status: 6 of 9 files reviewed, 2 unresolved discussions (waiting on @dorimedini-starkware).

---

crates/starknet_committer/src/db/index_db/leaves.rs line 39 at r9 (raw file):

Previously, dorimedini-starkware wrote…

1. reuse existing constant
2. if const doesn't work use LazyLock (see existing examples in our code)
3. not sure how you should be converting Felt to bytes, I just gave an example here; also you can consider just defining Felts in this context and converting to bytes only in the db_prefix method

Done.

3 won't work since I need a reference for the non-dynamic prefixes (the whole point of Cow and keeping the static lifetime), but I can take slices of a [u8;32] under a lazy lock.

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 33 at r8 (raw file):

Previously, dorimedini-starkware wrote…

1. please add some cases that indicate the byte order: i.e. larger values
2. how is it possible that the ContractState case serializes into just 3 bytes...? don't you need to measure the byte length of a value to know how to deserialize it? if the actual byte size of the three values are 10, 10 and 12, you will get 32 bytes, and try to deserialize as a binary node..?

Re 2, leaves are not mixed with inner nodes due to DeserializationContext that has is_leaf.

Re 1, added two non-trivial thresholds.

[dorimedini-starkware]

@dorimedini-starkware reviewed 3 files and all commit messages, made 3 comments, and resolved 2 discussions.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @ArielElp).

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 45 at r10 (raw file):

        128_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8, 0_u8,
        0_u8, 0_u8,
    ])))

works? if so, much clearer

Suggestion:

    IndexLayoutStarknetStorageValue(StarknetStorageValue(Felt::from_bytes_be(&[
            vec![0_u8; 15], vec![128_u8], vec![0_u8; 16]
        ].concat()
    ])))

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 65 at r10 (raw file):

// 27 nibbles, the chooser is the number of bytes. In this case, the first byte will be 11000000
// (chooser 12, i.e. we need 12 bytes) followed by the value.
#[case(starknet_storage_value_leaf_96_bits(), DbValue(vec![192, 128, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]))]

Suggestion:

DbValue([vec![192, 128], vec![0; 11]].concat())

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 74 at r10 (raw file):

    0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0
]))]

Suggestion:

#[case(starknet_storage_value_leaf_136_bits(), DbValue([
    vec![240], vec![0; 14], vec![128], vec![0; 16]
].concat()))]

[nimrod-starkware]

@nimrod-starkware partially reviewed 1 file and made 1 comment.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @ArielElp and @yoavGrs).

---

crates/starknet_committer/src/db/index_db/leaves.rs line 186 at r10 (raw file):

    for felt in felts {
        felt.serialize(&mut buffer)?;
    }

@yoavGrs needed because of this

Code quote:

fn serialize_felts(felts: &[Felt]) -> Result<DbValue, SerializationError> {
    let mut buffer = Vec::new();
    for felt in felts {
        felt.serialize(&mut buffer)?;
    }

[ArielElp]

@ArielElp made 3 comments.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @dorimedini-starkware and @yoavGrs).

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 45 at r10 (raw file):

Previously, dorimedini-starkware wrote…

works? if so, much clearer

done differently but solves the issue

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 65 at r10 (raw file):

// 27 nibbles, the chooser is the number of bytes. In this case, the first byte will be 11000000
// (chooser 12, i.e. we need 12 bytes) followed by the value.
#[case(starknet_storage_value_leaf_96_bits(), DbValue(vec![192, 128, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]))]

Done.

---

crates/starknet_committer/src/db/index_db/leaves_test.rs line 74 at r10 (raw file):

    0, 0, 0, 0, 0, 0, 0, 0,
    0, 0, 0, 0, 0, 0, 0, 0
]))]

Done.

[dorimedini-starkware]

@dorimedini-starkware reviewed 1 file and all commit messages, and resolved 3 discussions.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @ArielElp and @yoavGrs).

[ArielElp]

@ArielElp resolved 1 discussion.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ArielElp).