tw/tcp

[rob-maron]

No description provided.

[bfish713]

I didn't look through all the cliqnet code because it was reviewed in the other repo, but the ConnectedNetwork impl and changes look good

[sveitser]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request introduces a new, secure, and robust networking layer called cliquenet for the HotShot framework. It establishes a fully connected mesh network using TCP and the Noise protocol for authenticated communication. The integration updates core networking traits and their implementations to leverage ViewNumber and EpochNumber for message context, and extends the testing suite to validate the new network's functionality.

Highlights

• New Cliquenet Network Implementation: Introduced a new cliquenet crate that provides a secure, authenticated, fully connected mesh network layer built on TCP and the Noise protocol. This includes modules for addressing, channels, error handling, framing, message IDs, network core logic, retry mechanisms, time utilities, and X25519 cryptography.
• HotShot Integration and Testing: Integrated the cliquenet network into the HotShot framework by adding CliquenetImpl to hotshot-example-types and updating the ConnectedNetwork trait and its implementations across hotshot-types and hotshot to support the new network. This also involved updating numerous test files to include CliquenetImpl in cross-tests.
• ViewNumber and EpochNumber in Network Traits: Modified the ConnectedNetwork trait methods (e.g., broadcast_message, direct_message, da_broadcast_message, vid_broadcast_message, update_view) to explicitly include ViewNumber and EpochNumber parameters, enhancing message handling and context within the network.
• Dependency Updates: Updated Cargo.lock and Cargo.toml files to incorporate the new cliquenet crate and its associated dependencies, as well as updating versions for several existing cryptographic and utility crates.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• Cargo.lock
  • Added new dependencies for cliquenet and its cryptographic components, including aead, aes, alloca, bon, chacha20, chacha20poly1305, cipher, cliquenet, ct-codecs, ed25519-compact, env_logger, ghash, inout, nohash-hasher, page_size, poly1305, polyval, quickcheck, snow, and universal-hash.
  • Updated versions for various existing dependencies to accommodate the new additions and ensure compatibility.
• Cargo.toml
  • Added crates/cliquenet to the members and default-members lists.
  • Included bimap, bytes, bon, cliquenet (with metrics feature), nohash-hasher, and quickcheck as new dependencies.
• crates/cliquenet/Cargo.toml
  • New file: Defines the cliquenet crate, specifying its version, edition, description, features (metrics), and dependencies.
• crates/cliquenet/README.md
  • New file: Provides a detailed overview of the cliquenet networking layer, covering its use of TCP and Noise protocol, architecture (server tasks, connect tasks, handshakes, I/O tasks), simultaneous connection handling, heartbeats, latency measurements, and channel communication.
• crates/cliquenet/benches/bench1.rs
  • New file: Implements benchmarks to compare the performance of raw TCP communication against the cliquenet implementation for various message sizes.
• crates/cliquenet/src/addr.rs
  • New file: Defines the Address enum for representing network addresses (IP or hostname) and provides methods for parsing, serialization, and port manipulation.
• crates/cliquenet/src/chan.rs
  • New file: Implements a custom channel (Sender, Receiver) that ensures only a single copy of an item (identified by an Id) is kept, useful for deduplication.
• crates/cliquenet/src/error.rs
  • New file: Defines the NetworkError enum, encompassing various errors that can occur within the cliquenet networking layer, such as I/O errors, invalid headers, Noise protocol errors, and timeouts.
• crates/cliquenet/src/frame.rs
  • New file: Defines the Header structure for network frames, detailing its components (Version, Type, Partial flag, Reserved bits, Payload length) and providing methods for creation and parsing.
• crates/cliquenet/src/id.rs
  • New file: Defines a simple Id struct for uniquely identifying messages, implementing Hash and nohash_hasher::IsEnabled.
• crates/cliquenet/src/lib.rs
  • New file: The main library file for cliquenet, exporting public modules, defining NetConf for network configuration, Role for peer roles, and constants like MAX_MESSAGE_SIZE.
• crates/cliquenet/src/metrics.rs
  • New file: Provides NetworkMetrics for collecting and reporting various network-related metrics, including connection attempts, latency, and queue capacities for peers.
• crates/cliquenet/src/net.rs
  • New file: Implements the core Network logic, including the main server loop, handling of inbound and outbound connections, Noise protocol handshakes, I/O task management, and message routing (unicast, multicast, broadcast).
• crates/cliquenet/src/retry.rs
  • New file: Implements a Retry mechanism for messages, wrapping the Network to provide acknowledgements, message buffering, and retransmission logic based on buckets and message IDs.
• crates/cliquenet/src/time.rs
  • New file: Defines Timestamp for microsecond-precision time measurements and Countdown for resettable timers, used for heartbeats and timeouts.
• crates/cliquenet/src/x25519.rs
  • New file: Provides wrappers for X25519 Keypair, PublicKey, and SecretKey using the ed25519-compact library, along with associated error types.
• crates/cliquenet/tests/frame-handling.rs
  • New file: Contains tests for cliquenet's ability to handle multiple frames and various message sizes, ensuring correct message assembly and delivery.
• crates/hotshot/example-types/src/membership/fetcher.rs
  • Modified DirectMessageFn and direct_message_impl to accept ViewNumber as a parameter, providing more context for direct messages.
  • Updated Leaf2Fetcher to pass the ViewNumber when sending leaf responses and requests.
• crates/hotshot/example-types/src/node_types.rs
  • Added CliquenetImpl struct and implemented NodeImplementation for it, associating it with the new Cliquenet network type.
• crates/hotshot/hotshot/Cargo.toml
  • Added cliquenet as a dependency with the metrics feature enabled.
• crates/hotshot/hotshot/src/lib.rs
  • Modified the da_broadcast_message call to include view_number.u64().into() as a parameter.
• crates/hotshot/hotshot/src/traits.rs
  • Added Cliquenet to the implementations module, making it available as a network type.
• crates/hotshot/hotshot/src/traits/networking.rs
  • Added a new module cliquenet_network and re-exported Cliquenet from it.
• crates/hotshot/hotshot/src/traits/networking/cliquenet_network.rs
  • New file: Implements the Cliquenet struct as a ConnectedNetwork for HotShot, integrating the cliquenet::Retry mechanism. Includes derive_keypair for key generation and TestableNetworkingImplementation for testing purposes.
• crates/hotshot/hotshot/src/traits/networking/combined_network.rs
  • Updated broadcast_message, da_broadcast_message, direct_message, vid_broadcast_message, and update_view methods to accept ViewNumber and EpochNumber objects instead of raw u64 values.
• crates/hotshot/hotshot/src/traits/networking/libp2p_network.rs
  • Updated broadcast_message, da_broadcast_message, direct_message, and update_view methods to accept ViewNumber and EpochNumber objects.
• crates/hotshot/hotshot/src/traits/networking/memory_network.rs
  • Updated broadcast_message, da_broadcast_message, and direct_message methods to accept ViewNumber.
• crates/hotshot/hotshot/src/traits/networking/push_cdn_network.rs
  • Updated broadcast_message, da_broadcast_message, and direct_message methods to accept ViewNumber.
• crates/hotshot/hotshot/src/types/handle.rs
  • Modified calls to broadcast_message, direct_message, and da_broadcast_message to pass the ViewNumber extracted from the message.
• crates/hotshot/task-impls/src/network.rs
  • Modified vid_broadcast_message to store ViewNumber alongside the message data.
  • Updated update_view to pass ViewNumber and EpochNumber objects.
  • Modified calls to direct_message, broadcast_message, and da_broadcast_message to pass the ViewNumber.
• crates/hotshot/testing/src/byzantine/byzantine_behaviour.rs
  • Modified direct_message calls within byzantine behaviors to pass the ViewNumber.
• crates/hotshot/testing/tests/*.rs
  • Updated various test files (e.g., test_epoch_end.rs, test_epoch_success_catchup_types.rs, test_epoch_success_overlap_2_f.rs, test_epoch_success_overlap_2f_plus_1.rs, test_epoch_success_overlap_3f.rs, test_epoch_success_overlap_dynamic.rs, test_epoch_success_overlap_f.rs, test_epoch_success_overlap_f_plus_1.rs, test_epoch_success_types.rs, test_epoch_success_types_randomized_leader.rs, test_epoch_unequal_stake.rs, test_epochs_failures.rs, test_shorter_decide.rs, test_success_with_async_delay_2_with_epochs.rs, test_success_with_async_delay_with_epochs.rs, test_success_with_epochs.rs, test_vid2_success.rs, test_with_double_leader_no_failures_with_epochs.rs, tests_1/test_success.rs, tests_1/test_with_failures_2.rs, tests_3/byzantine_tests.rs, tests_3/memory_network.rs, tests_3/test_with_failures_half_f.rs, tests_4/test_with_failures_f.rs, tests_5/test_with_failures.rs) to include CliquenetImpl in their cross_tests! macros, enabling testing with the new network implementation.
• crates/hotshot/types/src/data.rs
  • Added From<u64> for $t and From<$t> for u64 implementations for ViewNumber, EpochNumber, and other u64 wrapper types.
  • Corrected comments related to QuorumProposal2 legacy versions.
• crates/hotshot/types/src/traits/network.rs
  • Updated the ConnectedNetwork trait methods (broadcast_message, da_broadcast_message, vid_broadcast_message, direct_message, update_view, queue_node_lookup) to explicitly include ViewNumber and EpochNumber parameters, providing more granular control and context for network operations.
• node-metrics/src/api/node_validator/v0/mod.rs
  • Removed an unnecessary comment.
• request-response/src/network.rs
  • Removed the blanket Deref implementation for Sender and related ConnectedNetwork trait methods, simplifying the Sender trait definition.
• sequencer/src/external_event_handler.rs
  • Updated OutboundMessage::Direct and OutboundMessage::Broadcast handling to extract and pass ViewNumber to the network's direct_message and broadcast_message calls.

Activity

• rob-maron created the pull request.
• sveitser requested a summary of the pull request.