apollo_propeller: add tree topology core logic

[sirandreww-starkware]

No description provided.

[sirandreww-starkware]

• apollo_propeller: add minimal config #11196 : 2 dependent PRs (#11197 , #11942 )
• apollo_propeller: add behaviour skeleton #11195
• apollo_propeller: add Event enum to types #11194
• apollo_propeller: add dial upgrade error handling to handler #11074
• apollo_propeller: add rate-limited queue length warnings #11073
• apollo_propeller: add concurrent streams support to handler #11072
• apollo_propeller: implement outbound substream with message batching #11071
• apollo_propeller: add handler helper methods #11070
• apollo_propeller: use protobuf type in handler send queue #11069
• apollo_propeller: reorder handler structs #11068
• apollo_propeller: add inbound substream handling to receive units #11067
• apollo_propeller: add receive queue and emit logic to handler #11066
• apollo_propeller: add handler skeleton with TODOs #11065
• apollo_propeller: add protocol upgrade for libp2p streams #11064
• apollo_protobuf: add prost codec for length-delimited messages #11063
• apollo_propeller: add protobuf conversions for PropellerUnit #11062
• apollo_propeller: add merkle proof verification to validator #11061
• apollo_propeller: integrate tree origin verification in validator #11060
• apollo_propeller: add tree origin verification methods #11059
• apollo_propeller: add unit validator with signature caching #11058
• apollo_propeller: add tree topology core logic #11057 👈 (View in Graphite)
• apollo_propeller: add shard validation error type #11056
• apollo_propeller: add peer set error type #11055
• apollo_propeller: add core unit struct with accessors #11054
• apollo_propeller: add signature verification module #11053
• apollo_propeller: add reconstruction error type #11052
• apollo_propeller: add shard publish error type #11051
• apollo_propeller: add tree generation error type #11050
• apollo_propeller: add ShardSignatureVerificationError #11049
• apollo_propeller: add base types (Channel, ShardIndex, MessageRoot) #11048
• apollo_propeller: add reed-solomon erasure coding #11047
• apollo_propeller: added merkle implementation tests #11206 : 1 other dependent PR (#11572 )
• apollo_propeller: added merkle tree builder #11205
• apollo_propeller: added model merkle implementation #10985
• apollo_propeller: added model merkle proof implementation #11204
• apollo_protobuf: added propeller proto file #10654
• apollo_propeller: new crate for the propeller p2p protocol #11192
• main-v0.14.1-committer

This stack of pull requests is managed by Graphite. Learn more about stacking.

[reviewable-StarkWare]

This change is 

[sirandreww-starkware]

@sirandreww-starkware made 11 comments and resolved 4 discussions.
Reviewable status: 1 of 2 files reviewed, 8 unresolved discussions (waiting on @guy-starkware, @noamsp-starkware, and @ShahakShama).

---

crates/apollo_propeller/src/tree.rs line 23 at r1 (raw file):

Previously, ShahakShama wrote…

Rename Tree to Schedule or Router

Done.

---

crates/apollo_propeller/src/tree.rs line 25 at r1 (raw file):

Previously, ShahakShama wrote…

+1

Decided to go with a sorted vector for efficiency

---

crates/apollo_propeller/src/tree.rs line 25 at r1 (raw file):

Previously, ShahakShama wrote…

I thought this is a channel. Rename it to channel and consider adding a dedicated type / type alias

Done.

---

crates/apollo_propeller/src/tree.rs line 50 at r1 (raw file):

Previously, ShahakShama wrote…

Just because we wrote it on the board as f doesn't mean in the code it should be f as well. Rename to num_data_shards

Done.

---

crates/apollo_propeller/src/tree.rs line 53 at r1 (raw file):

Previously, ShahakShama wrote…

Why? I want propeller to work with 1 node as well

Added a TODO

---

crates/apollo_propeller/src/tree.rs line 69 at r1 (raw file):

Previously, ShahakShama wrote…

Add comment that we deduct the publisher

Done.

---

crates/apollo_propeller/src/tree.rs line 84 at r1 (raw file):

Previously, ShahakShama wrote…

cluster nodes?

Done.

---

crates/apollo_propeller/src/tree.rs line 86 at r1 (raw file):

Previously, ShahakShama wrote…

Rename to move_to_new_channel

I actually want to change this struct to be immutable after it is created. I want to do this because I want to tie this with the fact the same channel should not change the set inside of it once it is created. I moved this code to new. Please take another look

---

crates/apollo_propeller/src/tree.rs line 108 at r1 (raw file):

Previously, ShahakShama wrote…

+1

Done

---

crates/apollo_propeller/src/tree.rs line 162 at r1 (raw file):

Previously, guy-starkware wrote…

Seems superfluous.

Done

---

crates/apollo_propeller/src/tree.rs line 62 at r1 (raw file):

    }

    pub fn calculate_data_shards(&self) -> usize {

Done

[guy-starkware]

@guy-starkware reviewed 3 files and all commit messages, and resolved 1 discussion.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @noamsp-starkware and @ShahakShama).

[ShahakShama]

@ShahakShama reviewed 1 file and all commit messages, made 7 comments, and resolved 6 discussions.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @noamsp-starkware and @sirandreww-starkware).

---

crates/apollo_propeller/src/tree.rs line 23 at r1 (raw file):

Previously, sirandreww-starkware (Andrew Luka) wrote…

Done.

rename the file as well (or add a TODO)

---

crates/apollo_propeller/src/tree.rs line 25 at r2 (raw file):

pub struct PropellerScheduleManager {
    /// All nodes in the channel with their stake, sorted by (stake, peer_id) descending.
    channel_nodes: Vec<(PeerId, Stake)>,

I tend to think it's better to not store the Stake as long as we don't do anything with it

---

crates/apollo_propeller/src/tree.rs line 38 at r2 (raw file):

impl PropellerScheduleManager {
    /// Create a new propeller tree manager.

comment is outdated. I think it's unneccesary to begin with and can be deleted

---

crates/apollo_propeller/src/tree.rs line 60 at r2 (raw file):

        let total_nodes = nodes.len();
        // Ensure num_data_shards is at least 1 for small networks (N=2,3)
        // Standard formula: num_data_shards = floor((N-1)/3)

Add the comment here that you reduce N by 1 because you exclude the publisher

---

crates/apollo_propeller/src/tree_test.rs line 7 at r2 (raw file):

#[test]
fn test_create_empty_tree_manager() {

Rename tree across this file too

---

crates/apollo_propeller/src/tree_test.rs line 21 at r2 (raw file):

#[test]
fn test_update_nodes_and_f_calculation() {

rename f here
Also rename update_nodes

---

crates/apollo_propeller/src/tree_test.rs line 63 at r2 (raw file):

#[test]
fn test_update_nodes_missing_local_peer() {

rename update_nodes to new

[sirandreww-starkware]

@sirandreww-starkware made 7 comments.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @noamsp-starkware and @ShahakShama).

---

crates/apollo_propeller/src/tree.rs line 23 at r1 (raw file):

Previously, ShahakShama wrote…

rename the file as well (or add a TODO)

Done.

---

crates/apollo_propeller/src/tree.rs line 25 at r2 (raw file):

Previously, ShahakShama wrote…

I tend to think it's better to not store the Stake as long as we don't do anything with it

I would rather change this implementation to the "build = 1/3 stake" than go backwards here. Would you allow that in this PR?

---

crates/apollo_propeller/src/tree.rs line 38 at r2 (raw file):

Previously, ShahakShama wrote…

comment is outdated. I think it's unneccesary to begin with and can be deleted

Done.

---

crates/apollo_propeller/src/tree.rs line 60 at r2 (raw file):

Previously, ShahakShama wrote…

Add the comment here that you reduce N by 1 because you exclude the publisher

Done.

---

crates/apollo_propeller/src/tree_test.rs line 7 at r2 (raw file):

Previously, ShahakShama wrote…

Rename tree across this file too

Done.

---

crates/apollo_propeller/src/tree_test.rs line 21 at r2 (raw file):

Previously, ShahakShama wrote…

rename f here
Also rename update_nodes

Done.

---

crates/apollo_propeller/src/tree_test.rs line 63 at r2 (raw file):

Previously, ShahakShama wrote…

rename update_nodes to new

Done.

[ShahakShama]

@ShahakShama reviewed 4 files and all commit messages, made 1 comment, and resolved 7 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @noamsp-starkware).

[ShahakShama]

@ShahakShama reviewed 1 file and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @noamsp-starkware).

[sirandreww-starkware]

@sirandreww-starkware reviewed 4 files and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @noamsp-starkware).