Merge branch 'main' into conflicting-txs

Author: SupernaviX

Logbook.md

@@ -1,5 +1,40 @@
 # Leios logbook
 
+## 2025-06-17
+
+### Mini-mainnet experiments
+
+The 750-node [pseudo-mainnet](data/simulation/pseudo-mainnet/topology-v2.yaml) network was used in Haskell and Rust experiments to study the limits of transaction and IB throughput for realistic scenarios up to 300 TPS and 32 IB/s.
+
+- [Analysis results](analysis/sims/2025w24/analysis.ipynb)
+- [Slides](analysis/sims/2025w24/summary.pdf)
+- Findings:
+    - The 750 node mini-mainnet is a suitable replacement for the 10,000-node pseudo mainnet, in that either topology would result in similar performance measurements and resource recommendations.
+    - The Haskell and Rust simulations substantially agree for mini-mainnet simulations.
+    - Key metrics from these simulations:
+        - Block propagation less than 1 second, which is consistent with empirical observations from pooltool.io. Note that this has implications for our discussion of the IB-concurrency period.
+        - With 1 Gb/s links/NICs, the protocol can support 25 MB/s throughput before it starts degrading.
+        - Mean time from mempool to ledger is about 150 seconds for transactions.
+        - Disk-space efficiency is about 80%.
+        - About 20% of network traffic is wasted.
+        - Even at 300 ts/x, a 6-core VM is sufficient for peak demand, but average demand is less than 2 cores.
+
+### Added features to simulation trace processor
+
+The [`leios-trace-processor`](analysis/sims/trace-processor/) now reports message sizes for bandwidth-usage analysis.
+
+### Pseudo-mainnet experiments
+
+The 10,000-node [pseudo-mainnet](data/simulation/pseudo-mainnet/ReadMe.md) network was used in Rust experiments to study the limits of transaction and IB throughput for realistic scenarios up to 300 TPS and 32 IB/s.
+
+- [Analysis results](analysis/sims/2025w23/analysis.ipynb)
+- [Slides](analysis/sims/2025w23/summary.pdf)
+- Findings:
+    - Transactions took an average of 100 seconds to travel from the memory pool to the ledger.
+    - Disk and network usage was approximately 80% efficient.
+    - Even at high TPS, six CPU cores were sufficient to handle peak load.
+    - Block propagation time averaged under one second.
+
 ## 2025-06-15
 
 ### Reduced memory footprint for analyzing simulation traces

analysis/sims/.gitignore

@@ -2,11 +2,13 @@
 .ipynb_checkpoints/
 ols
 sim-cli
+leios-trace-processor
 tmp/
 plots/
 results/
 stdout
 stderr
 *.csv.gz
 *.log.gz
+*.log.xz
 index.html

analysis/sims/2025w23/analysis.ipynb

@@ -0,0 +1,33 @@
+#!/usr/bin/env nix-shell
+#!nix-shell -i bash -p gzip
+
+set -e
+
+mkdir -p results/{tps3x,ibs}
+
+for d in ibs tps3x
+do
+  for f in cpus lifecycle resources receipts
+  do
+    DIR=$(find $d -type f -name $f.csv.gz \( -not -empty \) -printf %h\\n -quit)
+    HL=$(sed -n -e '1p' "$DIR/case.csv")
+    HR=$(zcat "$DIR/$f.csv.gz" | sed -n -e '1p')
+    (
+      echo "$HL,$HR"
+      for g in $(find $d -type f -name $f.csv.gz \( -not -empty \) -printf %h\\n)
+      do
+        if [ ! -e "$g/stderr" ]
+        then
+          echo "Skipping $g because it has no stderr." >> /dev/stderr
+        elif [ -s "$g/stderr" ]
+        then
+          echo "Skipping $g because its stderr is not empty." >> /dev/stderr
+        else
+          BL=$(sed -n -e '2p' "$g/case.csv")
+          zcat "$g/$f.csv.gz" | sed -e "1d;s/^/$BL,/;s/null/NA/g"
+        fi
+      done
+    ) | gzip -9c > results/$d/$f.csv.gz &
+  done
+done
+wait

analysis/sims/2025w23/combine-results.sh

@@ -0,0 +1 @@
+6e7e493b682602d4df70157aac309bff0ca9b64e

analysis/sims/2025w23/git.hash

@@ -0,0 +1,2 @@
+simulator,ibps
+rust,0.5

analysis/sims/2025w23/ibs/rust-0.5/case.csv

@@ -0,0 +1,217 @@
+# yaml-language-server: $schema=./config.schema.json
+
+################################################################################
+# Simulation Configuration File
+################################################################################
+#
+# This file contains the default configuration for running Leios simulations in
+# the Haskell simulation (`simulation/`) and the Rust simulation (`sim-rs/`).
+#
+################################################################################
+# Simulation Configuration
+################################################################################
+
+relay-strategy: "request-from-first"
+tcp-congestion-control: true
+multiplex-mini-protocols: true
+simulate-transactions: false
+treat-blocks-as-full: false
+cleanup-policies: ["cleanup-expired-vote"]
+
+################################################################################
+# Leios Protocol Configuration
+################################################################################
+
+leios-variant: full
+leios-stage-length-slots: 10
+leios-stage-active-voting-slots: 1
+leios-vote-send-recv-stages: false
+leios-late-ib-inclusion: true
+leios-header-diffusion-time-ms: 1000.0
+# TODO: revise default
+praos-chain-quality: 100
+praos-fallback-enabled: false
+
+################################################################################
+# Transaction Configuration
+################################################################################
+
+tx-generation-distribution:
+  distribution: exp
+  lambda: 0.85
+  scale: 800.0
+tx-size-bytes-distribution:
+  distribution: log-normal
+  mu: 6.833
+  sigma: 1.127
+tx-sharded-percentage: 1.0
+tx-validation-cpu-time-ms: 1.5
+tx-max-size-bytes: 16384
+tx-start-time: 300
+tx-stop-time: 300
+
+################################################################################
+# Ranking Block Configuration
+################################################################################
+
+# 1/leios-stage-length-slots, targeting one RB per pipeline.
+# Also 20s is current rate of praos blocks.
+rb-generation-probability: 5.0e-2
+# Eng. team targets 1kB as worst case upper bound.
+# Actual size fairly close.
+rb-head-size-bytes: 1024
+rb-body-max-size-bytes: 90112
+# Note: certificate generation/validation is not included in the
+# timings here, see cert-* fields.
+rb-generation-cpu-time-ms: 1.0
+rb-head-validation-cpu-time-ms: 1.0
+
+# On average, no Txs directly embedded in blocks.
+rb-body-legacy-praos-payload-avg-size-bytes: 0
+rb-body-legacy-praos-payload-validation-cpu-time-ms-constant: 50.0
+# the -per-byte component is meant to be using size as a (bad)
+# proxy for the complexity of the Txs included.
+rb-body-legacy-praos-payload-validation-cpu-time-ms-per-byte: 0.0005
+
+################################################################################
+# Input Block Configuration
+################################################################################
+
+ib-generation-probability: 0.5
+ib-shards: 1
+ib-shard-period-length-slots: 10
+ib-shard-group-count: 1
+
+# ProducerId  32
+# SlotNo      64
+# VRF proof   80
+# Body hash   32
+# RB Ref      32
+# Signature   64
+# Total       304
+#
+# NOTE: using a KES Signature (like for Praos headers)
+#       would instead more than double the total to 668.
+#       And even 828 including Op Cert.
+ib-head-size-bytes: 304
+# 98KB to optimize for 3 TCP round trips
+ib-body-avg-size-bytes: 196608
+ib-body-max-size-bytes: 196608
+# Here we also use praos blocks as ballpark estimate.
+# Sec 2.3 Forging, of the benchmark cluster report, lists
+#   * Slot start to announced: 0.12975s
+ib-generation-cpu-time-ms: 130.0
+ib-head-validation-cpu-time-ms: 1.0
+ib-body-validation-cpu-time-ms-constant: 50.0
+ib-body-validation-cpu-time-ms-per-byte: 0.0005
+ib-diffusion-strategy: "freshest-first"
+
+# Haskell prototype relay mini-protocol parameters.
+ib-diffusion-max-bodies-to-request: 1
+ib-diffusion-max-headers-to-request: 100
+ib-diffusion-max-window-size: 100
+
+################################################################################
+# Endorsement Block Configuration
+################################################################################
+
+# We want one per pipeline, but not too many.
+eb-generation-probability: 2.5
+# ProducerId  32
+# SlotNo      64
+# VRF proof   80
+# Signature   64
+# Total       240
+#
+# See Note about signatures on ib-head-size-bytes.
+eb-size-bytes-constant: 240
+# IB hash
+eb-size-bytes-per-ib: 32
+# Collecting the IBs to reference and cryptography are the main tasks.
+# A comparable task is maybe mempool snapshotting.
+# Sec 2.3 Forging, of the benchmark cluster report, lists
+#   * Mempool snapshotting: 0.07252s
+# 75ms then seems a generous estimate for eb generation.
+eb-generation-cpu-time-ms: 75.0
+# Validating signature and vrf proof, as in other headers.
+eb-validation-cpu-time-ms: 1.0
+
+eb-diffusion-strategy: "peer-order"
+
+# Haskell prototype relay mini-protocol parameters.
+eb-diffusion-max-bodies-to-request: 1
+eb-diffusion-max-headers-to-request: 100
+eb-diffusion-max-window-size: 100
+
+# The maximum age of EBs included in RBs.
+# A an EB from slot `s` can only be included in RBs
+# up to slot `s+eb-max-age-slots`.
+# In short leios we expect votes to diffuse within 3 stages lengths of
+# EB generation, we allow for 2 more stage lengths to account for
+# variance in the interval within RBs.
+eb-max-age-slots: 240
+
+# The maximum age of EBs to be relayed.
+# An EB from slot `s` will only be relayed
+# up to slot `s+eb-max-age-for-relay-slots`.
+eb-max-age-for-relay-slots: 40
+
+# The maximum size of transactions (in bytes) which an EB can reference.
+# Only relevant when running with the "full-without-ibs" variant.
+eb-referenced-txs-max-size-bytes: 16384000
+
+################################################################################
+# Vote Configuration
+################################################################################
+
+# Cryptography related values taken from [vote-spec](crypto-benchmarks.rs/Specification.md)
+# using weighted averages of 80% persistent and 20% non-persistent.
+
+# vote-spec#Committe and quorum size
+#
+# Note: this is used as the expected amount of total weight of
+# generated votes in the sims.
+vote-generation-probability: 500.0
+# vote-spec#"Committe and quorum size"
+# 60% of `vote-generation-probability`
+vote-threshold: 300
+# vote-spec#"Generate vote" 0.8*135e-3 + 0.2*280e-3
+vote-generation-cpu-time-ms-constant: 164.0e-3
+# No benchmark yet.
+vote-generation-cpu-time-ms-per-ib: 0
+# vote-spec#"Verify vote" 0.8*670e-3 + 0.2*1.4
+vote-validation-cpu-time-ms: 816.0e-3
+# The `Vote` structure counted in the -per-eb already identifies slot
+# (in Eid) and voter. We can assume a vote bundle is all for the same
+# voter and slot, so for non-persistent voters we could factor their
+# PoolKeyHash (28bytes) here, but that is for 20% of cases.
+# More relevant if EB generation is very high.
+vote-bundle-size-bytes-constant: 0
+# vote-spec#Votes 0.8*90 + 0.2*164
+vote-bundle-size-bytes-per-eb: 105
+
+vote-diffusion-strategy: "peer-order"
+
+# Haskell prototype relay mini-protocol parameters.
+vote-diffusion-max-bodies-to-request: 1
+vote-diffusion-max-headers-to-request: 100
+vote-diffusion-max-window-size: 100
+
+################################################################################
+# Certificate Configuration
+################################################################################
+
+# vote-spec - certificate size plot.
+# Realistic stake distributions need about 7 kilobytes for the certificate.
+cert-size-bytes-constant: 7168
+cert-size-bytes-per-node: 0
+
+# For certificate timings we have bulk figures for realistic scenarios,
+# so we do not attempt to give -per-node (i.e. per-voter) timings.
+#
+# vote-spec#"Generate certificate"
+cert-generation-cpu-time-ms-constant: 90.0
+cert-generation-cpu-time-ms-per-node: 0
+# vote-spec#"Verify certificate"
+cert-validation-cpu-time-ms-constant: 130.0
+cert-validation-cpu-time-ms-per-node: 0

analysis/sims/2025w23/ibs/rust-0.5/config.yaml

@@ -0,0 +1 @@
+../../topology-v1.yaml

analysis/sims/2025w23/ibs/rust-0.5/network.yaml

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+set -e
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+
+mkfifo sim.log
+
+grep -E -v '(Slot|No.*Generated|CpuTask|Sent|Lottery)' sim.log | xz -9c > sim.log.xz &
+
+../../sim-cli --parameters config.yaml network.yaml --slots 300 --conformance-events sim.log > stdout 2> stderr
+
+wait
+rm sim.log
+
+cat << EOI > case.csv
+simulator,tps
+rust,$(basename $PWD)
+EOI
+
+cat case.csv

analysis/sims/2025w23/ibs/rust-0.5/run.sh

@@ -0,0 +1,54 @@
+praos: 141 transactions(s) were generated in total.
+praos: 20 naive praos block(s) were published.
+praos: 280 slot(s) had no naive praos blocks.
+praos: 120 transaction(s) (23.59 MB) finalized in a naive praos block.
+praos: 21 transaction(s) (4.13 MB) did not reach a naive praos block.
+praos: Pool 1 published 1 naive praos block(s)
+praos: Pool 179 published 1 naive praos block(s)
+praos: Pool 668 published 1 naive praos block(s)
+praos: Pool 746 published 1 naive praos block(s)
+praos: Pool 4931 published 2 naive praos block(s)
+praos: Pool 5957 published 1 naive praos block(s)
+praos: Pool 6007 published 1 naive praos block(s)
+praos: Pool 6010 published 1 naive praos block(s)
+praos: Pool 6031 published 1 naive praos block(s)
+praos: Pool 6257 published 1 naive praos block(s)
+praos: Pool 6272 published 1 naive praos block(s)
+praos: Pool 6315 published 1 naive praos block(s)
+praos: Pool 6357 failed to publish 1 naive praos block(s) due to slot battles.
+praos: Pool 6512 published 1 naive praos block(s)
+praos: Pool 6829 published 1 naive praos block(s)
+praos: Pool 6843 published 1 naive praos block(s)
+praos: Pool 6877 published 1 naive praos block(s)
+praos: Pool 6908 published 1 naive praos block(s)
+praos: Pool 7446 published 1 naive praos block(s)
+praos: Pool 7930 published 1 naive praos block(s)
+leios: 141 IB(s) were generated, on average 0.470 IB(s) per slot.
+leios: 141 out of 141 transaction(s) were included in at least one IB.
+leios: The average age of the pending transactions is 23.048s (stddev 14.049).
+leios: Each transaction was included in an average of 1.000 IB(s) (stddev 0.000).
+leios: Each IB contained an average of 1.000 transaction(s) (stddev 0.000) and an average of 196.91 kB (stddev 0 B). 0 IB(s) were empty.
+leios: Each node received an average of 140.998 IB(s) (stddev 0.063).
+leios: 72 EB(s) were generated; on average there were 0.240 EB(s) per slot.
+leios: Each EB contained an average of 17.833 IB(s) (stddev 4.619). 0 EB(s) were empty.
+leios: Each IB was included in an average of 10.110 EB(s) (stddev 3.407).
+leios: 127 out of 141 IBs were included in at least one EB.
+leios: 0 out of 141 IBs expired before they reached an EB.
+leios: 5 out of 72 EBs expired before an EB from their stage reached an RB.
+leios: 127 out of 141 transaction(s) were included in at least one EB.
+leios: 115107 total votes were generated.
+leios: Each stake pool produced an average of 43.322 vote(s) (stddev 91.276).
+leios: Each EB received an average of 1692.750 vote(s) (stddev 618.995).
+leios: There were 7937 bundle(s) of votes. Each bundle contained 14.503 vote(s) (stddev 15.538).
+leios: 17 L1 block(s) had a Leios endorsement.
+leios: 120 tx(s) (23.59 MB) were referenced by a Leios endorsement.
+leios: 0 tx(s) (0 B) were included directly in a Praos block.
+leios: Spatial efficiency: 23.59 MB/27.95 MB (84.415%) of Leios bytes were unique transactions.
+leios: 326 tx(s) (73.094%) referenced by a Leios endorsement were redundant.
+leios: Each transaction took an average of 0.130s (stddev 0.000) to be included in an IB.
+leios: Each transaction took an average of 25.610s (stddev 3.243) to be included in an EB.
+leios: Each transaction took an average of 60.449s (stddev 12.460) to be included in a block.
+network: 0 TX message(s) were sent. 0 of them were received (NaN%).
+network: 1409836 IB message(s) were sent. 1409835 of them were received (100.000%).
+network: 719928 EB message(s) were sent. 719928 of them were received (100.000%).
+network: 79362063 Vote message(s) were sent. 79362063 of them were received (100.000%).