Prototype to Measure Effect of EB Traffic on Praos

[dnadales]

As part of our progress on #544, we aim to measure the impact that EBs may have on the performance of the Praos protocol. The goal of this experiment is to test the hypothesis that Leios cannot run without disrupting Praos, and to measure the magnitude of this disruption (see EXP-LeiosDiffusionOnly in the Impact Analysis document).

Experiment Setup

We plan to configure an experiment where:

• A Cardano node is connected to one upstream mock peer and one downstream peer.
• The Cardano node processes messages from the upstream peer and, after appropriate processing, relays them to the downstream peer.

Run Configurations

Run 1:

• The upstream mock peer serves only Praos blocks.

Run 2:

• The upstream mock peer serves Praos blocks.
• The upstream mock peer also serves EB blocks and the transactions they reference.

Analysis

After both runs are completed, we should analyze the resulting node logs to generate plots and perform statistical analysis on the time it takes for a node to relay a Praos block. Specifically, we want to measure the duration from the moment the node receives a Praos block to the moment it forwards it to its downstream peer.

Constraints and Implementation Notes

• The Praos implementation must remain untouched for this experiment.
• The mempool should not be modified.
• We may optionally use the P&T tooling to analyze node logs, but it is not required.

Prototype Requirements

This prototype should include:

• A mini-protocol for EB diffusion.
• A transaction cache used by the process responsible for receiving and relaying blocks. This cache will be queried to request missing transactions referenced in EB blocks.
• We need some nontrivial constraints on the bandwidth and latency of the connection to the mocked upstream peer. The Innovation team can provide us with some representative numbers.

EB Handling Logic

• Upon receiving an EB, the node will request any missing transactions from the tx cache via the mocked upstream peer.
• There will be no transaction submission traffic between these nodes.
• The transaction cache will be used to avoid redundant requests when multiple EBs reference the same transactions.

Additional Notes

• We could use a trace from the Leios simulator to mock the data.
• The components described in this issue are a subset of those outlined in the Impact Analysis document.

Extensions to the prototype

The prototype described in this issue could be extended in the following ways (work which is outside of the scope of this issue):

• Add more nodes under tests.
• Add EB votes.

[nfrisby]

Here's my current notes about what the patched node would need for the demo sketched above, at least the Minimum Viable Demo (MVD) version of it.

I'm sharing this as a Comment, since it has a lot of details, many of which will likely change during our work on this. But I think it'll also help our colleagues to already have a clearer picture of what we're targeting right now.

This notes will eventually evolve into better explanations in a demo design document---but it doesn't already exist!

• Only EB offer/req/rsp and EB tx offer/req/rsp messages. No votes, no EB announcements, and no range requests.
• TxCache, so that overlapping EBs don't cause redundant Tx requests
• EbStore, so that the node can serve its acquired EBs to downstream peers in whatever order they request them
• mini protocol client logic + centralized decision logic (who to fetch what from—simply only the first to offer, in the MVD)
• mini protocol server and very simple "follower" for EbStore, so that it can send appropriate EB offers and EB tx offers.
• mocked upstream peer serving both the pre-scheduled Praos blocks and pre-scheduled Leios blocks
• Sources/generators for those pre-scheduled blocks
• We could use libfaketime or something like that, but it actually doesn't matter that the Praos blocks are all old when they arrive to the node under test, as long as they have competitive BlockNos. (Until we start using age-based eviction policies. Then the wall clock would matter, at least for the most robust implementation.)
• mocked downstream peer requesting only offers from the node under test (including MsgRequestNext), and recording arrival timestamps of those delivered offers.
• Some means to setup and tear down the patched node under test and the two mocked peers, along with some non-trivial bandwidth and latency constraints on their network connections (an easy-but-not-very-reproducible option: we simply run them on machines in different physical locations). cardano-ignite etc hopefully makes sense for larger iterations of this demo.

Fundamental observation:

• The delay between the timestamp when the mocked upstream peer first sends the MsgRollForward for each Praos block and the timestamp of when the mocked downstream peer first receives the same MsgRollForward for that block from the node-under-test.
• We can repeat the demo multiple times with varying Leios loads in order to see how it affects the fundamental (Praos) observation.

Secondary observation:

• Whether the mocked downstream peer is able to receive the EBs’ closures, as proof that the node under test is actually acquiring them from the mock upstream peer.
• (We don’t need that for the Praos blocks, since the patched node’s Praos code is untouched.)

[ch1bo]

@dnadales I added this as-is to the leios roadmap. Please update input-output-hk/ouroboros-leios#544 to only cover what will not yet be done by this (let's avoid another hierarchy level).

[nfrisby]

@dnadales @bladyjoker Here's my thoughts on decomposing this Issue into some initial tasks that are all quite independent.

• Write a little Haskell library for statically generating the pre-scheduled Leios traffic.
• Somehow acquire a small non-trivial Praos chain (and the corresponding config files); circa 50 blocks that are closer to full than to empty. (I'm guessing the leftovers of a P&T cluster run would be the easiest way to acquire this.) Just having one would be fine to get the ball rolling.
• Write the mocked upstream peer (probably as a patch of the immdb-server) that can replay that Praos chain by dripfeeding it as the wall clock advances while simultaneously dripfeeding the statically generated Leios blocks as per their schedule.
• Define the mini protocol syntax and codecs. (edit: see Prototype to Measure Effect of EB Traffic on Praos #1701 (comment) below)
• Write the EbStore and the TxCache. My gut so far has said a single sqlite db is a totally appropriate place to start, for both stores. Fully on disk, and no eviction at all, to start with.
• Write the mini protocol client logic and the Leios fetch logic. This would response to notifications, request EBs, read from the TxCache, request EB txs, write to the TxCache and the EbStore.
• Scripts for setup and tear down (including eg the node's topology config files). Could be as simple as ~30 line bash script, I'm guessing.

Next steps:

• Write the mocked downstream peer, just as an implementation-independent event logger, basically, although it can also test for correctness by confirming that the node-under-test can actually correctly serve the Leios blocks.
• Write the (much simpler) mini protocol server logic. This would pull from the EbStore as needed, and it would also send corresponding notifications whenever the Leios fetch logic writes to the EbStore.

There will be more steps after that, but I think that's a good set to consider for initial work towards the skeleton of the demo.

[nfrisby]

FYI, I've sketched up the (first version of the) mini protocols on this branch: https://github.com/IntersectMBO/ouroboros-consensus/tree/nfrisby/issue-1701-mini-protocols

The commit(s) there branch off from the ouroboros-consensus commit used by the latest node release, https://github.com/IntersectMBO/cardano-node/releases/tag/10.5.1, so hopefully it'll be very easy to produce a runnable patched node, once the stubs are filled in.

Just remember: since it's a first stab at code for the demo prototype, it's not at all tested.

[sterraf]

FYI, I've sketched up the (first version of the) mini protocols on this branch: https://github.com/IntersectMBO/ouroboros-consensus/tree/nfrisby/issue-1701-mini-protocols

@nfrisby Could you point me to some example myManifest.json for using with leiosdemo202510 generate?

[nfrisby]

@sterraf I replied on this bespoke PR #1730

If anyone else has any questions about the WIP demo code, please ask on that PR instead of on this Issue (this Issue isn't intended to be "user facing" so-to-speak).