apollo_protobuf: add prost codec for length-delimited messages

[sirandreww-starkware]

No description provided.

[reviewable-StarkWare]

This change is 

[sirandreww-starkware]

• apollo_propeller: add minimal config #11196 : 2 dependent PRs (#11197 , #11942 )
• apollo_propeller: add behaviour skeleton #11195
• apollo_propeller: add Event enum to types #11194
• apollo_propeller: add dial upgrade error handling to handler #11074
• apollo_propeller: add rate-limited queue length warnings #11073
• apollo_propeller: add concurrent streams support to handler #11072
• apollo_propeller: implement outbound substream with message batching #11071
• apollo_propeller: add handler helper methods #11070
• apollo_propeller: use protobuf type in handler send queue #11069
• apollo_propeller: reorder handler structs #11068
• apollo_propeller: add inbound substream handling to receive units #11067
• apollo_propeller: add receive queue and emit logic to handler #11066
• apollo_propeller: add handler skeleton with TODOs #11065
• apollo_propeller: add protocol upgrade for libp2p streams #11064
• apollo_protobuf: add prost codec for length-delimited messages #11063 👈 (View in Graphite)
• apollo_propeller: add protobuf conversions for PropellerUnit #11062
• apollo_propeller: add merkle proof verification to validator #11061
• apollo_propeller: integrate tree origin verification in validator #11060
• apollo_propeller: add tree origin verification methods #11059
• apollo_propeller: add unit validator with signature caching #11058
• apollo_propeller: add tree topology core logic #11057
• apollo_propeller: add shard validation error type #11056
• apollo_propeller: add peer set error type #11055
• apollo_propeller: add core unit struct with accessors #11054
• apollo_propeller: add signature verification module #11053
• apollo_propeller: add reconstruction error type #11052
• apollo_propeller: add shard publish error type #11051
• apollo_propeller: add tree generation error type #11050
• apollo_propeller: add ShardSignatureVerificationError #11049
• apollo_propeller: add base types (Channel, ShardIndex, MessageRoot) #11048
• apollo_propeller: add reed-solomon erasure coding #11047
• apollo_propeller: added merkle implementation tests #11206 : 1 other dependent PR (#11572 )
• apollo_propeller: added merkle tree builder #11205
• apollo_propeller: added model merkle implementation #10985
• apollo_propeller: added model merkle proof implementation #11204
• apollo_protobuf: added propeller proto file #10654
• apollo_propeller: new crate for the propeller p2p protocol #11192
• main-v0.14.1-committer

This stack of pull requests is managed by Graphite. Learn more about stacking.

[sirandreww-starkware]

@sirandreww-starkware made 4 comments.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @noamsp-starkware and @ShahakShama).

---

crates/apollo_protobuf/src/codec_test.rs line 77 at r4 (raw file):

Previously, ShahakShama wrote…

No. I meant that i is equal to the last byte of a message

I really don't understand. Are you on the latest version of the PR? Can you give a psudo-code example?

---

crates/apollo_protobuf/src/codec_test.rs line 138 at r4 (raw file):

Previously, ShahakShama wrote…

Where is it?

I don't understand this? Is this a case that is not covered by the byte by byte implementation?

---

crates/apollo_protobuf/src/codec_test.rs line 38 at r5 (raw file):

Previously, ShahakShama wrote…

20ms is too much (think that if we have 10000 of tests which there's a chance we have, it will take 200 seconds to run the tests if the bar for all of them is 20ms)

1. Reduce test_multiple_messages_byte_by_byte constant from 15 to let's say 10
2. Remove the 10k case from test_encode_decode_multiple_messages_all_at_once::case_6
3. Remove the 100k case from test_encode_decode_various_sizes

Done.

---

crates/apollo_protobuf/src/codec_test.rs line 56 at r6 (raw file):

Previously, ShahakShama wrote…

Don't have the same message all over the test. Also, consider a changing message len. For example, message len = i % 100

This change will probably require you to have a smaller num_messages, which is ok. I prefer an interesting test over a big test

Done.

[ShahakShama]

@ShahakShama reviewed 3 files and all commit messages, made 2 comments, and resolved 3 discussions.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @noamsp-starkware and @sirandreww-starkware).

---

crates/apollo_protobuf/src/codec_test.rs line 77 at r4 (raw file):

Previously, sirandreww-starkware (Andrew Luka) wrote…

I really don't understand. Are you on the latest version of the PR? Can you give a psudo-code example?

Here's an ugly version of what I want. Please prettify it and add it

let mut expected_next_message_index_in_full_buf = sent_messages.first().unwrap().len();
for i in 0..total_len {
    ...
    if partial_buf.is_empty() {
        assert_eq!(i, expected_next_message_index_in_full_buf);
        expected_next_message_index_in_full_buf += sent_messages[received_messages + 1].len();
        ...

---

crates/apollo_protobuf/src/codec_test.rs line 138 at r4 (raw file):

Previously, sirandreww-starkware (Andrew Luka) wrote…

I don't understand this? Is this a case that is not covered by the byte by byte implementation?

Oh right

[sirandreww-starkware]

@sirandreww-starkware made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @noamsp-starkware and @ShahakShama).

---

crates/apollo_protobuf/src/codec_test.rs line 77 at r4 (raw file):

Previously, ShahakShama wrote…

Here's an ugly version of what I want. Please prettify it and add it

let mut expected_next_message_index_in_full_buf = sent_messages.first().unwrap().len();
for i in 0..total_len {
    ...
    if partial_buf.is_empty() {
        assert_eq!(i, expected_next_message_index_in_full_buf);
        expected_next_message_index_in_full_buf += sent_messages[received_messages + 1].len();
        ...

Done.

[ShahakShama]

@ShahakShama reviewed 1 file and all commit messages, made 1 comment, and resolved 1 discussion.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @noamsp-starkware).

[sirandreww-starkware]

@sirandreww-starkware partially reviewed 9 files and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @noamsp-starkware).