starknet_committer,starknet_committer_and_os_cli,starknet_patricia: layout dependent fetch_nodes

[ArielElp]

No description provided.

[reviewable-StarkWare]

This change is 

[ArielElp]

• starknet_committer,starknet_patricia: move node_serde to facts db #11644
• starknet_committer: reorganize db modules #11640
• starknet_committer: add commit e2e tests #11331
• index layout forest creation tests #11286
• starknet_committer,starknet_patricia: layout-based original tree creation tests #11236
• starknet_committer: add original tree creation test templates #11715
• starknet_committer,starknet_patricia: layout-dependent db-key separator #11641
• starknet_committer,starknet_patricia: remove FilledNode impl #11637
• starknet_committer: db layout trait #11621
• starknet_committer,starknet_patricia: layout-dependent FilledTree serialization #10988
• starknet_committer,starknet_patricia: refactor key context #11592
• starknet_committer,starknet_patricia: add layout for base trait #11566
• starknet_committer,starknet_patricia: add serialization utilities to node layout #10987
• starknet_committer,starknet_patricia: move db_layout #10986
• starknet_committer: add index db #10685
• starknet_committer: abstract original tree creation #10684
• starknet_committer: enrich layout trait to abstract skeleton creation #11182
• starknet_committer: move trie creation logic to trie_traversal #11180
• starknet_committer: group generic layout logic #10852
• starknet_committer,starknet_patricia: remove generic config #10683
• starknet_committer: add index layout nodes #10682
• starknet_committer: add index layout subtree #10919
• starknet_committer: hash leaves of both layouts #10918
• starknet_committer: add index layout leaves #10681
• starknet_committer,starknet_patricia,starknet_patricia_storage: support contract address prefix #10858
• starknet_committer,starknet_patricia: add serde errors #10810
• starknet_committer,starknet_committer_and_os_cli,starknet_patricia: layout dependent fetch_nodes #10680 👈 (View in Graphite)
• starknet_committer,starknet_patricia: enrich subtree trait to support fetch nodes #10751
• starknet_committer,starknet_committer_and_os_cli,starknet_patricia: make FilledNode generic #10750
• starknet_committer,starknet_os,starknet_patricia: make node data generic #10679
• starknet_committer,starknet_patricia: add deserialization context #10704
• starknet_committer,starknet_patricia,starknet_patricia_storage: add context to prefix and serde #10678
• starknet_committer,starknet_patricia: add subtree trait #10694
• starknet_committer: reorganize facts db files #10677
• main-v0.14.1-committer

This stack of pull requests is managed by Graphite. Learn more about stacking.

[yoavGrs]

@yoavGrs reviewed 2 files and all commit messages, made 2 comments, and resolved 4 discussions.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @ArielElp and @nimrod-starkware).

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 273 at r9 (raw file):

Previously, ArielElp wrote…

Diverged even further, ptal at handle_unmodified_subtree

I missed the meaning of the call to get_bottom_subtree.
I guess this function is non-trivial and involves heavy computation as well.

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 262 at r12 (raw file):

    if let NodeData::Edge(EdgeData { bottom_data, path_to_bottom }) = filled_root.data {
        // Even if a subtree rooted at an edge node is umodified, we still need an

Suggestion:

unmodified

[yoavGrs]

@yoavGrs made 1 comment and resolved 1 discussion.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @ArielElp and @nimrod-starkware).

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 262 at r12 (raw file):

    if let NodeData::Edge(EdgeData { bottom_data, path_to_bottom }) = filled_root.data {
        // Even if a subtree rooted at an edge node is umodified, we still need an

Edge case :)

[nimrod-starkware]

@nimrod-starkware reviewed all commit messages, made 1 comment, and resolved 1 discussion.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @ArielElp).

[nimrod-starkware]

please open py side

@nimrod-starkware made 1 comment.
Reviewable status: 13 of 14 files reviewed, 2 unresolved discussions (waiting on @ArielElp, @dorimedini-starkware, and @yoavGrs).

[dorimedini-starkware]

yes, py side please :)

@dorimedini-starkware reviewed 5 files and all commit messages, and made 8 comments.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @ArielElp).

---

crates/starknet_committer/src/db/facts_db/db.rs line 46 at r13 (raw file):

use crate::patricia_merkle_tree::types::CompiledClassHash;

pub struct FactsNodeLayout {}

docstring describing the storage scheme in high-level?
or is there a more natural location for this doc?

Code quote:

pub struct FactsNodeLayout {}

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 64 at r13 (raw file):

    L: Leaf,
    Layout: NodeLayout<'a, L>,
    FilledNode<L, Layout::NodeData>: DBObject<DeserializeContext = Layout::DeserializationContext>,

hmmmm don't you have

impl<L: Leaf, Layout: NodeLayout> DBObject<DeserializeContext = Layout::DeserializationContext> for FilledNode<L, Layout::NodeData>

that prevents the need for this bound?

Code quote:

FilledNode<L, Layout::NodeData>: DBObject<DeserializeContext = Layout::DeserializationContext>

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 70 at r13 (raw file):

    let should_fetch_modified_leaves =
        config.compare_modified_leaves() || previous_leaves.is_some();
    while !current_subtrees.is_empty() {

this swap is technically a logic change that should be in a separate PR, because I had to scan to make sure that previous_leaves is not mutated inside the loop. I see it isn't, so ok, but try to keep this in mind when creating PRs

Code quote:

    let should_fetch_modified_leaves =
        config.compare_modified_leaves() || previous_leaves.is_some();
    while !current_subtrees.is_empty() {

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 254 at r13 (raw file):

    subtree: SubTree,
) where
    SubTree::NodeData: Copy,

please be explicit when a copy is needed

Suggestion:

SubTree::NodeData: Clone,

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 282 at r13 (raw file):

            .nodes
            .insert(root_index, OriginalSkeletonNode::UnmodifiedSubTree(filled_root.hash));
    }

1. please use an explicit match on the three options of filled_root.data, instead of if let .. else
2. previously, an unmodified leaf just triggered a warning; now, you seem to be pushing it to the skeleton tree. why?

Code quote:

    if let NodeData::Edge(EdgeData { bottom_data, path_to_bottom }) = filled_root.data {
        // Even if a subtree rooted at an edge node is unmodified, we still need an
        // `OriginalSkeletonNode::Edge` node in the skeleton in case we need to manipulate it later
        // (e.g. unify the edge node with an ancestor edge node).
        skeleton_tree.nodes.insert(root_index, OriginalSkeletonNode::Edge(path_to_bottom));
        match SubTree::should_traverse_unmodified_child(bottom_data) {
            UnmodifiedChildTraversal::Traverse => {
                let (bottom_subtree, _) = subtree.get_bottom_subtree(&path_to_bottom, bottom_data);
                next_subtrees.push(bottom_subtree);
            }
            UnmodifiedChildTraversal::Skip(unmodified_child_hash) => {
                skeleton_tree.nodes.insert(
                    path_to_bottom.bottom_index(root_index),
                    OriginalSkeletonNode::UnmodifiedSubTree(unmodified_child_hash),
                );
            }
        }
    } else {
        skeleton_tree
            .nodes
            .insert(root_index, OriginalSkeletonNode::UnmodifiedSubTree(filled_root.hash));
    }

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 295 at r13 (raw file):

) {
    let is_leaf = subtree.is_leaf();
    let is_modified = !subtree.is_unmodified();

are either of these used more than once? seems like a redundant definition

Code quote:

    let is_leaf = subtree.is_leaf();
    let is_modified = !subtree.is_unmodified();

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 309 at r13 (raw file):

        }
        return;
    }

this entire block of logic is basically equivalent to the if we had before - please revert the "unrolling" so it's easier to see the diff

Suggestion:

    let is_modified = !subtree.is_unmodified();
    if !subtree.is_leaf() || (should_fetch_modified_leaves && is_modified) {
        next_subtrees.push(subtree);
    } else if !is_modified {
        match SubTree..
    }

[ArielElp]

@ArielElp made 6 comments and resolved 2 discussions.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @dorimedini-starkware).

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 64 at r13 (raw file):

Previously, dorimedini-starkware wrote…

hmmmm don't you have

impl<L: Leaf, Layout: NodeLayout> DBObject<DeserializeContext = Layout::DeserializationContext> for FilledNode<L, Layout::NodeData>

that prevents the need for this bound?

There is a constraint on the layout trait, hence using it requires satisfying the constraint here.

FWIW, the FilledTree part of the stack adds an associated NodeDbObject: DbObject to the layout trait, so these requirements may be unnecessary then.

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 254 at r13 (raw file):

Previously, dorimedini-starkware wrote…

please be explicit when a copy is needed

WDYM?

Clone indeed allows more freedom, but in practice our types here are () and HashOutput, both copyable. If I require Clone, then bottom_data will need to be cloned in:

match SubTree::should_traverse_unmodified_child(bottom_data.clone()) {

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 282 at r13 (raw file):

Previously, dorimedini-starkware wrote…

1. please use an explicit match on the three options of filled_root.data, instead of if let .. else
2. previously, an unmodified leaf just triggered a warning; now, you seem to be pushing it to the skeleton tree. why?

1. Done
2. This is related to the last case of handle_child_subtree.

Before: an unmodified leaf was always added to the skeleton upon encountering its parent, it was never traversed itself (hence the "unexpected" warning)
Now: when we traverse unmodified children, we will go through this leaf in the main loop, so it's no longer unexpected.

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 295 at r13 (raw file):

Previously, dorimedini-starkware wrote…

are either of these used more than once? seems like a redundant definition

No, they're there for clarity

is_modified was Yoav's suggestion IIRC to make the control flow more clear. IMO it does make it more clear (no "not unmodified" in the if condition).

is_leaf is less important so removed

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 309 at r13 (raw file):

Previously, dorimedini-starkware wrote…

this entire block of logic is basically equivalent to the if we had before - please revert the "unrolling" so it's easier to see the diff

Slight pushback: I think the control flow is much clearer now

Internal --> traverse
Modified --> if should_fetch_modified_leaves then traverse
Unmodified --> new logic

---

crates/starknet_committer/src/db/facts_db/db.rs line 46 at r13 (raw file):

Previously, dorimedini-starkware wrote…

docstring describing the storage scheme in high-level?
or is there a more natural location for this doc?

Done, this is probably the most reasonable place

[ArielElp]

python side successful
https://github.com/starkware-industries/starkware/pull/39782

@ArielElp made 1 comment.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @dorimedini-starkware).

[dorimedini-starkware]

WOOP!

@dorimedini-starkware reviewed 2 files and all commit messages, made 3 comments, and resolved 5 discussions.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @ArielElp).

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 254 at r13 (raw file):

Previously, ArielElp wrote…

WDYM?

Clone indeed allows more freedom, but in practice our types here are () and HashOutput, both copyable. If I require Clone, then bottom_data will need to be cloned in:

match SubTree::should_traverse_unmodified_child(bottom_data.clone()) {

that's good IMO, it's the same just explicit; when data is copied I'd like to know

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 309 at r13 (raw file):

Previously, ArielElp wrote…

Slight pushback: I think the control flow is much clearer now

Internal --> traverse
Modified --> if should_fetch_modified_leaves then traverse
Unmodified --> new logic

kibalti

[ArielElp]

@ArielElp made 1 comment.
Reviewable status: 12 of 14 files reviewed, 1 unresolved discussion (waiting on @dorimedini-starkware and @yoavGrs).

---

crates/starknet_committer/src/db/facts_db/create_facts_tree.rs line 254 at r13 (raw file):

Previously, dorimedini-starkware wrote…

that's good IMO, it's the same just explicit; when data is copied I'd like to know

Done, note that I also changed the constraint in the layout so there are a couple more clones in this file.

[dorimedini-starkware]

@dorimedini-starkware reviewed 2 files and all commit messages, and resolved 1 discussion.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ArielElp).

[dorimedini-starkware]

@dorimedini-starkware made 1 comment.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ArielElp).

[nimrod-starkware]

@nimrod-starkware reviewed 1 file and all commit messages, and made 1 comment.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ArielElp).

[dorimedini-starkware]

@dorimedini-starkware reviewed 1 file and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ArielElp).