apollo_network_benchmark: add message index detection mechanism#11557
Conversation
|
This change is |
|
There hasn't been any activity on this pull request recently, and in order to prioritize active work, it has been marked as stale. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| let mut index_tracker = vec![MessageIndexTracker::default(); num_peers]; | ||
| let mut all_pending = 0; | ||
| while let Some((peer_id, index)) = rx.recv().await { | ||
| let old_pending = index_tracker[peer_id].pending_messages_count(); |
There was a problem hiding this comment.
Sender ID indexing can panic receiver
High Severity
record_indexed_message indexes index_tracker by sender_id, but the vector is sized with bootstrap.len(). This assumes sender IDs are dense zero-based indices, which NodeArgs.runner.id does not enforce. Valid deployments with sparse or non-zero-based IDs can trigger out-of-bounds access and crash the receiver path.
Additional Locations (1)
sirandreww-starkware
force-pushed
the
01-08-apollo_network_benchmark_add_message_index_detection_mechanism
branch
from
a581d33 to
c7ca4ed
Compare
sirandreww-starkware
force-pushed
the
01-08-apollo_network_benchmark_add_message_index_detection_mechanism
branch
from
c7ca4ed to
f10f148
Compare
sirandreww-starkware
force-pushed
the
01-08-apollo_network_benchmark_add_messageindextracker_struct
branch
from
0794786 to
b05e636
Compare
| } | ||
| .boxed() | ||
| let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); | ||
| let num_peers = self.args.runner.bootstrap.len(); |
There was a problem hiding this comment.
Index tracker sized by bootstrap peers, indexed by sender_id
High Severity
num_peers is set to self.args.runner.bootstrap.len(), which is the number of other peers (N−1 for N nodes). But record_indexed_message uses this to size index_tracker and indexes it by sender_id (which is runner.id, ranging from 0 to N−1). For the node with the highest ID, index_tracker[N-1] is out of bounds on a vec of length N−1, causing a panic at runtime.
Additional Locations (1)
sirandreww-starkware
force-pushed
the
01-08-apollo_network_benchmark_add_message_index_detection_mechanism
branch
from
f10f148 to
6bed8fb
Compare
sirandreww-starkware
force-pushed
the
01-08-apollo_network_benchmark_add_messageindextracker_struct
branch
from
b05e636 to
41a1832
Compare
sirandreww-starkware
force-pushed
the
01-08-apollo_network_benchmark_add_messageindextracker_struct
branch
from
41a1832 to
3093d1d
Compare
sirandreww-starkware
force-pushed
the
01-08-apollo_network_benchmark_add_message_index_detection_mechanism
branch
from
6bed8fb to
2186fdd
Compare
2 participants
Note
Medium Risk
Adds new async coordination between the receive path and an index-tracking task via an unbounded channel, which can impact runtime behavior and memory under high load. Also changes the
receive_stress_test_messagecallback signature and receiver task wiring, which is easy to get wrong and could break compilation or message processing if misused.Overview
Adds message index gap detection to the broadcast network stress test node by sending
(sender_id, message_index)fromreceive_stress_test_messageinto a newrecord_indexed_messagetask that maintains per-peerMessageIndexTrackerstate.Introduces a new gauge metric,
RECEIVE_MESSAGE_PENDING_COUNT, and updates the node to run two receive-side tasks (receiver + index tracker) instead of a single receiver task, so dashboards can observe how many messages are still “missing” based on observed indices.Written by Cursor Bugbot for commit 2186fdd. This will update automatically on new commits. Configure here.