Bandwidth experiment at tag leios-2025w25 (#418)

* Designed bandwidth experiment
* Fixed bug in message size retrieval
* Analysis template for bandwidth experiment
* 100 TPS experiment
* Analysis of 300 tps case
* Updated logbook

Author: bwbush

Logbook.md

@@ -1,5 +1,17 @@
 # Leios logbook
 
+## 2025-06-24
+
+### Bandwidth experiment
+
+Experiments exploring the effect of bandwidth limits at 100 TPS and 300 TPS were completed in [analysis/sims/2025w25/analysis.ipynb](analysis/sims/2025w25/analysis.ipynb). Key findings follow:
+
+1. The following protocol parameters are sufficient for Leios high performance at 100 tx/s (or 300 tx/s): 1 IB/s (or 2 IB/s), 10 slot/stage, 328 kB/IB maximum, 1.5 EB/stage, and multiple shards.
+2. Spatial efficiency is 80%.
+3. All transactions reach the ledger, typically within two minutes.
+4. A 30 Mbps network interface card (NIC) is sufficient for the Leios node.
+5. A 4-core vCPU is also sufficient.
+
 ## 2025-06-20
 
 ### CDDL Specification Draft

analysis/sims/2025w25/analysis.ipynb

@@ -0,0 +1,217 @@
+# yaml-language-server: $schema=./config.schema.json
+
+################################################################################
+# Simulation Configuration File
+################################################################################
+#
+# This file contains the default configuration for running Leios simulations in
+# the Haskell simulation (`simulation/`) and the Rust simulation (`sim-rs/`).
+#
+################################################################################
+# Simulation Configuration
+################################################################################
+
+relay-strategy: "request-from-first"
+tcp-congestion-control: true
+multiplex-mini-protocols: true
+simulate-transactions: true
+treat-blocks-as-full: false
+cleanup-policies: ["cleanup-expired-vote"]
+
+################################################################################
+# Leios Protocol Configuration
+################################################################################
+
+leios-variant: full
+leios-stage-length-slots: 10
+leios-stage-active-voting-slots: 1
+leios-vote-send-recv-stages: false
+leios-late-ib-inclusion: true
+leios-header-diffusion-time-ms: 1000.0
+# TODO: revise default
+praos-chain-quality: 100
+praos-fallback-enabled: false
+
+################################################################################
+# Transaction Configuration
+################################################################################
+
+tx-generation-distribution:
+  distribution: exp
+  lambda: 0.85
+  scale:  8.891
+tx-size-bytes-distribution:
+  distribution: log-normal
+  mu: 6.833
+  sigma: 1.127
+tx-sharded-percentage: 1.0
+tx-validation-cpu-time-ms: 1.5
+tx-max-size-bytes: 16384
+tx-start-time: 60
+tx-stop-time: 660
+
+################################################################################
+# Ranking Block Configuration
+################################################################################
+
+# 1/leios-stage-length-slots, targeting one RB per pipeline.
+# Also 20s is current rate of praos blocks.
+rb-generation-probability: 5.0e-2
+# Eng. team targets 1kB as worst case upper bound.
+# Actual size fairly close.
+rb-head-size-bytes: 1024
+rb-body-max-size-bytes: 90112
+# Note: certificate generation/validation is not included in the
+# timings here, see cert-* fields.
+rb-generation-cpu-time-ms: 1.0
+rb-head-validation-cpu-time-ms: 1.0
+
+# On average, no Txs directly embedded in blocks.
+rb-body-legacy-praos-payload-avg-size-bytes: 0
+rb-body-legacy-praos-payload-validation-cpu-time-ms-constant: 50.0
+# the -per-byte component is meant to be using size as a (bad)
+# proxy for the complexity of the Txs included.
+rb-body-legacy-praos-payload-validation-cpu-time-ms-per-byte: 0.0005
+
+################################################################################
+# Input Block Configuration
+################################################################################
+
+ib-generation-probability: 1.00
+ib-shards: 60
+ib-shard-period-length-slots: 1
+ib-shard-group-count: 6
+
+# ProducerId  32
+# SlotNo      64
+# VRF proof   80
+# Body hash   32
+# RB Ref      32
+# Signature   64
+# Total       304
+#
+# NOTE: using a KES Signature (like for Praos headers)
+#       would instead more than double the total to 668.
+#       And even 828 including Op Cert.
+ib-head-size-bytes: 304
+# 98KB to optimize for 3 TCP round trips
+ib-body-avg-size-bytes: 98304
+ib-body-max-size-bytes: 327680
+# Here we also use praos blocks as ballpark estimate.
+# Sec 2.3 Forging, of the benchmark cluster report, lists
+#   * Slot start to announced: 0.12975s
+ib-generation-cpu-time-ms: 130.0
+ib-head-validation-cpu-time-ms: 1.0
+ib-body-validation-cpu-time-ms-constant: 50.0
+ib-body-validation-cpu-time-ms-per-byte: 0.0005
+ib-diffusion-strategy: "freshest-first"
+
+# Haskell prototype relay mini-protocol parameters.
+ib-diffusion-max-bodies-to-request: 1
+ib-diffusion-max-headers-to-request: 100
+ib-diffusion-max-window-size: 100
+
+################################################################################
+# Endorsement Block Configuration
+################################################################################
+
+# We want one per pipeline, but not too many.
+eb-generation-probability: 1.5
+# ProducerId  32
+# SlotNo      64
+# VRF proof   80
+# Signature   64
+# Total       240
+#
+# See Note about signatures on ib-head-size-bytes.
+eb-size-bytes-constant: 240
+# IB hash
+eb-size-bytes-per-ib: 32
+# Collecting the IBs to reference and cryptography are the main tasks.
+# A comparable task is maybe mempool snapshotting.
+# Sec 2.3 Forging, of the benchmark cluster report, lists
+#   * Mempool snapshotting: 0.07252s
+# 75ms then seems a generous estimate for eb generation.
+eb-generation-cpu-time-ms: 75.0
+# Validating signature and vrf proof, as in other headers.
+eb-validation-cpu-time-ms: 1.0
+
+eb-diffusion-strategy: "peer-order"
+
+# Haskell prototype relay mini-protocol parameters.
+eb-diffusion-max-bodies-to-request: 1
+eb-diffusion-max-headers-to-request: 100
+eb-diffusion-max-window-size: 100
+
+# The maximum age of EBs included in RBs.
+# A an EB from slot `s` can only be included in RBs
+# up to slot `s+eb-max-age-slots`.
+# In short leios we expect votes to diffuse within 3 stages lengths of
+# EB generation, we allow for 2 more stage lengths to account for
+# variance in the interval within RBs.
+eb-max-age-slots: 240
+
+# The maximum age of EBs to be relayed.
+# An EB from slot `s` will only be relayed
+# up to slot `s+eb-max-age-for-relay-slots`.
+eb-max-age-for-relay-slots: 40
+
+# The maximum size of transactions (in bytes) which an EB can reference.
+# Only relevant when running with the "full-without-ibs" variant.
+eb-referenced-txs-max-size-bytes: 16384000
+
+################################################################################
+# Vote Configuration
+################################################################################
+
+# Cryptography related values taken from [vote-spec](crypto-benchmarks.rs/Specification.md)
+# using weighted averages of 80% persistent and 20% non-persistent.
+
+# vote-spec#Committe and quorum size
+#
+# Note: this is used as the expected amount of total weight of
+# generated votes in the sims.
+vote-generation-probability: 500.0
+# vote-spec#"Committe and quorum size"
+# 60% of `vote-generation-probability`
+vote-threshold: 300
+# vote-spec#"Generate vote" 0.8*135e-3 + 0.2*280e-3
+vote-generation-cpu-time-ms-constant: 164.0e-3
+# No benchmark yet.
+vote-generation-cpu-time-ms-per-ib: 0
+# vote-spec#"Verify vote" 0.8*670e-3 + 0.2*1.4
+vote-validation-cpu-time-ms: 816.0e-3
+# The `Vote` structure counted in the -per-eb already identifies slot
+# (in Eid) and voter. We can assume a vote bundle is all for the same
+# voter and slot, so for non-persistent voters we could factor their
+# PoolKeyHash (28bytes) here, but that is for 20% of cases.
+# More relevant if EB generation is very high.
+vote-bundle-size-bytes-constant: 0
+# vote-spec#Votes 0.8*90 + 0.2*164
+vote-bundle-size-bytes-per-eb: 105
+
+vote-diffusion-strategy: "peer-order"
+
+# Haskell prototype relay mini-protocol parameters.
+vote-diffusion-max-bodies-to-request: 1
+vote-diffusion-max-headers-to-request: 100
+vote-diffusion-max-window-size: 100
+
+################################################################################
+# Certificate Configuration
+################################################################################
+
+# vote-spec - certificate size plot.
+# Realistic stake distributions need about 7 kilobytes for the certificate.
+cert-size-bytes-constant: 7168
+cert-size-bytes-per-node: 0
+
+# For certificate timings we have bulk figures for realistic scenarios,
+# so we do not attempt to give -per-node (i.e. per-voter) timings.
+#
+# vote-spec#"Generate certificate"
+cert-generation-cpu-time-ms-constant: 90.0
+cert-generation-cpu-time-ms-per-node: 0
+# vote-spec#"Verify certificate"
+cert-validation-cpu-time-ms-constant: 130.0
+cert-validation-cpu-time-ms-per-node: 0

analysis/sims/2025w25/bw-1IBps/1000/config.yaml

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+set -e
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+
+BW=$(basename $PWD)
+
+ulimit -S -m 24000000 -v 24000000
+
+mkfifo sim.log
+
+sed -e 's/"bandwidth-bytes-per-second":125000000/"bandwidth-bytes-per-second":'"$((125000 * BW))"'/g' \
+  ../../../../../data/simulation/pseudo-mainnet/topology-v2.yaml \
+  > network.yaml
+
+function cleanup() {
+  rm sim.log
+  rm network.yaml
+}
+trap cleanup EXIT
+
+grep -E -v '(Slot|No.*Generated|CpuTask|Lottery)' sim.log | pigz -p 3 -9c > sim.log.gz &
+
+../../sim-cli --parameters config.yaml network.yaml --slots 1200 --conformance-events sim.log > stdout 2> stderr
+
+wait
+
+cat << EOI > case.csv
+Simulator,Bandwidth [Mb/s]
+Rust,$(basename $PWD)
+EOI
+
+zcat sim.log.gz \
+| ../../leios-trace-processor \
+  +RTS -N5 -RTS \
+  --trace-file /dev/stdin \
+  --lifecycle-file lifecycle.csv \
+  --cpu-file cpus.csv \
+  --resource-file resources.csv \
+  --receipt-file receipts.csv
+  
+pigz -p 3 -9vf {lifecycle,cpus,resources,receipts}.csv
+
+cat case.csv

analysis/sims/2025w25/bw-1IBps/1000/run.sh

@@ -0,0 +1,2 @@
+Simulator,Bandwidth [Mb/s]
+Rust,12

analysis/sims/2025w25/bw-1IBps/12/case.csv

@@ -0,0 +1 @@
+../1000/config.yaml

analysis/sims/2025w25/bw-1IBps/12/config.yaml

@@ -0,0 +1 @@
+../1000/run.sh

analysis/sims/2025w25/bw-1IBps/12/run.sh

@@ -0,0 +1,82 @@
+praos: 59958 transactions(s) were generated in total.
+praos: 53 naive praos block(s) were published.
+praos: 1147 slot(s) had no naive praos blocks.
+praos: 59067 transaction(s) (100.36 MB) finalized in a naive praos block.
+praos: 891 transaction(s) (1.63 MB) did not reach a naive praos block.
+praos: Pool 1 published 0 naive praos block(s)
+praos: Pool 1 failed to publish 1 naive praos block(s) due to slot battles.
+praos: Pool 3 published 1 naive praos block(s)
+praos: Pool 43 published 1 naive praos block(s)
+praos: Pool 48 published 1 naive praos block(s)
+praos: Pool 57 published 1 naive praos block(s)
+praos: Pool 58 published 1 naive praos block(s)
+praos: Pool 63 published 1 naive praos block(s)
+praos: Pool 74 published 2 naive praos block(s)
+praos: Pool 89 published 1 naive praos block(s)
+praos: Pool 94 published 1 naive praos block(s)
+praos: Pool 95 published 1 naive praos block(s)
+praos: Pool 98 published 1 naive praos block(s)
+praos: Pool 103 published 1 naive praos block(s)
+praos: Pool 105 published 1 naive praos block(s)
+praos: Pool 119 published 1 naive praos block(s)
+praos: Pool 130 published 1 naive praos block(s)
+praos: Pool 131 published 1 naive praos block(s)
+praos: Pool 138 published 1 naive praos block(s)
+praos: Pool 361 published 1 naive praos block(s)
+praos: Pool 362 published 1 naive praos block(s)
+praos: Pool 371 published 2 naive praos block(s)
+praos: Pool 372 published 1 naive praos block(s)
+praos: Pool 377 published 1 naive praos block(s)
+praos: Pool 430 published 1 naive praos block(s)
+praos: Pool 434 published 1 naive praos block(s)
+praos: Pool 444 published 1 naive praos block(s)
+praos: Pool 447 published 1 naive praos block(s)
+praos: Pool 467 published 2 naive praos block(s)
+praos: Pool 474 published 2 naive praos block(s)
+praos: Pool 477 published 1 naive praos block(s)
+praos: Pool 478 published 2 naive praos block(s)
+praos: Pool 509 published 1 naive praos block(s)
+praos: Pool 518 published 1 naive praos block(s)
+praos: Pool 519 published 1 naive praos block(s)
+praos: Pool 536 published 1 naive praos block(s)
+praos: Pool 537 published 1 naive praos block(s)
+praos: Pool 546 published 1 naive praos block(s)
+praos: Pool 548 published 1 naive praos block(s)
+praos: Pool 552 published 2 naive praos block(s)
+praos: Pool 560 published 1 naive praos block(s)
+praos: Pool 561 published 1 naive praos block(s)
+praos: Pool 562 published 2 naive praos block(s)
+praos: Pool 565 published 1 naive praos block(s)
+praos: Pool 566 published 1 naive praos block(s)
+praos: Pool 742 published 1 naive praos block(s)
+praos: Pool 744 published 1 naive praos block(s)
+praos: Pool 746 published 1 naive praos block(s)
+leios: 1196 IB(s) were generated, on average 0.997 IB(s) per slot.
+leios: 59958 out of 59958 transaction(s) were included in at least one IB.
+leios: The average age of the pending transactions is 745.579s (stddev 65.828).
+leios: Each transaction was included in an average of 1.043 IB(s) (stddev 0.204).
+leios: Each IB contained an average of 52.309 transaction(s) (stddev 67.937) and an average of 89.31 kB (stddev 115). 522 IB(s) were empty.
+leios: Each node received an average of 1195.915 IB(s) (stddev 0.279).
+leios: 176 EB(s) were generated; on average there were 0.147 EB(s) per slot.
+leios: Each EB contained an average of 39.068 IB(s) (stddev 7.713). 0 EB(s) were empty.
+leios: Each IB was included in an average of 5.984 EB(s) (stddev 2.105).
+leios: 1149 out of 1196 IBs were included in at least one EB.
+leios: 12 out of 1196 IBs expired before they reached an EB.
+leios: 21 out of 176 EBs expired before an EB from their stage reached an RB.
+leios: 59067 out of 59958 transaction(s) were included in at least one EB.
+leios: 209633 total votes were generated.
+leios: Each stake pool produced an average of 970.523 vote(s) (stddev 200.568).
+leios: Each EB received an average of 1211.751 vote(s) (stddev 561.845).
+leios: There were 17303 bundle(s) of votes. Each bundle contained 12.115 vote(s) (stddev 16.319).
+leios: 51 L1 block(s) had a Leios endorsement.
+leios: 59067 tx(s) (100.36 MB) were referenced by a Leios endorsement.
+leios: 0 tx(s) (0 B) were included directly in a Praos block.
+leios: Spatial efficiency: 100.36 MB/107.54 MB (93.326%) of Leios bytes were unique transactions.
+leios: 232676 tx(s) (79.754%) referenced by a Leios endorsement were redundant.
+leios: Each transaction took an average of 66.141s (stddev 57.213) to be included in an IB.
+leios: Each transaction took an average of 95.637s (stddev 57.793) to be included in an EB.
+leios: Each transaction took an average of 133.181s (stddev 62.836) to be included in a block.
+network: 44908542 TX message(s) were sent. 44908542 of them were received (100.000%).
+network: 895759 IB message(s) were sent. 895740 of them were received (99.998%).
+network: 131824 EB message(s) were sent. 131824 of them were received (100.000%).
+network: 12959947 Vote message(s) were sent. 12959947 of them were received (100.000%).

analysis/sims/2025w25/bw-1IBps/12/summary.txt

@@ -0,0 +1,2 @@
+Simulator,Bandwidth [Mb/s]
+Rust,16

analysis/sims/2025w25/bw-1IBps/16/case.csv

@@ -0,0 +1 @@
+../1000/config.yaml