Skip to content

Commit d9d6639

Browse files
authored
Mini-mainnet experiments at tag leios-2025w24 (#403)
* Mini-mainnet experiments at tag `leios-2025w24` * Designed IBS experiments * Executed experiments * Data analysis of experiments * Summary slides * Updated logbook
1 parent 12a6943 commit d9d6639

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

100 files changed

+9597
-0
lines changed

Logbook.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,23 @@
22

33
## 2025-06-17
44

5+
### Mini-mainnet experiments
6+
7+
The 750-node [pseudo-mainnet](data/simulation/pseudo-mainnet/topology-v2.yaml) network was used in Haskell and Rust experiments to study the limits of transaction and IB throughput for realistic scenarios up to 300 TPS and 32 IB/s.
8+
9+
- [Analysis results](analysis/sims/2025w24/analysis.ipynb)
10+
- [Slides](analysis/sims/2025w24/summary.pdf)
11+
- Findings:
12+
- The 750 node mini-mainnet is a suitable replacement for the 10,000-node pseudo mainnet, in that either topology would result in similar performance measurements and resource recommendations.
13+
- The Haskell and Rust simulations substantially agree for mini-mainnet simulations.
14+
- Key metrics from these simulations:
15+
- Block propagation less than 1 second, which is consistent with empirical observations from pooltool.io. Note that this has implications for our discussion of the IB-concurrency period.
16+
- With 1 Gb/s links/NICs, the protocol can support 25 MB/s throughput before it starts degrading.
17+
- Mean time from mempool to ledger is about 150 seconds for transactions.
18+
- Disk-space efficiency is about 80%.
19+
- About 20% of network traffic is wasted.
20+
- Even at 300 ts/x, a 6-core VM is sufficient for peak demand, but average demand is less than 2 cores.
21+
522
### Added features to simulation trace processor
623

724
The [`leios-trace-processor`](analysis/sims/trace-processor/) now reports message sizes for bandwidth-usage analysis.

analysis/sims/2025w24/analysis.ipynb

Lines changed: 3688 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/usr/bin/env nix-shell
2+
#!nix-shell -i bash -p gzip
3+
4+
set -e
5+
6+
mkdir -p results/{tps,ibs}
7+
8+
for d in ibs tps
9+
do
10+
for f in cpus lifecycle resources receipts
11+
do
12+
DIR=$(find $d -type f -name $f.csv.gz \( -not -empty \) -printf %h\\n -quit)
13+
HL=$(sed -n -e '1p' "$DIR/case.csv")
14+
HR=$(zcat "$DIR/$f.csv.gz" | sed -n -e '1p')
15+
(
16+
echo "$HL,$HR"
17+
for g in $(find $d -type f -name $f.csv.gz \( -not -empty \) -printf %h\\n)
18+
do
19+
if [ ! -e "$g/stderr" ]
20+
then
21+
echo "Skipping $g because it has no stderr." >> /dev/stderr
22+
elif [ -s "$g/stderr" ]
23+
then
24+
echo "Skipping $g because its stderr is not empty." >> /dev/stderr
25+
else
26+
BL=$(sed -n -e '2p' "$g/case.csv")
27+
zcat "$g/$f.csv.gz" | sed -e "1d;s/^/$BL,/;s/null/NA/g"
28+
fi
29+
done
30+
) | gzip -9c > results/$d/$f.csv.gz &
31+
done
32+
done
33+
wait

analysis/sims/2025w24/ibs.list

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
ibs/rust-0.5/run.sh
2+
ibs/haskell-0.5/run.sh
3+
ibs/rust-1.0/run.sh
4+
ibs/haskell-1.0/run.sh
5+
ibs/rust-2.0/run.sh
6+
ibs/haskell-2.0/run.sh
7+
ibs/rust-4.0/run.sh
8+
ibs/haskell-4.0/run.sh
9+
ibs/rust-8.0/run.sh
10+
ibs/haskell-8.0/run.sh
11+
ibs/haskell-16.0/run.sh
12+
ibs/rust-16.0/run.sh
13+
ibs/haskell-32.0/run.sh
14+
ibs/rust-32.0/run.sh
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Simulator,IBPS
2+
Haskell,0.5
Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
# yaml-language-server: $schema=./config.schema.json
2+
3+
################################################################################
4+
# Simulation Configuration File
5+
################################################################################
6+
#
7+
# This file contains the default configuration for running Leios simulations in
8+
# the Haskell simulation (`simulation/`) and the Rust simulation (`sim-rs/`).
9+
#
10+
################################################################################
11+
# Simulation Configuration
12+
################################################################################
13+
14+
relay-strategy: "request-from-first"
15+
tcp-congestion-control: true
16+
multiplex-mini-protocols: true
17+
simulate-transactions: false
18+
treat-blocks-as-full: false
19+
cleanup-policies: ["cleanup-expired-vote"]
20+
21+
################################################################################
22+
# Leios Protocol Configuration
23+
################################################################################
24+
25+
leios-variant: full
26+
leios-stage-length-slots: 10
27+
leios-stage-active-voting-slots: 1
28+
leios-vote-send-recv-stages: false
29+
leios-late-ib-inclusion: true
30+
leios-header-diffusion-time-ms: 1000.0
31+
# TODO: revise default
32+
praos-chain-quality: 100
33+
praos-fallback-enabled: false
34+
35+
################################################################################
36+
# Transaction Configuration
37+
################################################################################
38+
39+
tx-generation-distribution:
40+
distribution: exp
41+
lambda: 0.85
42+
scale: 800.0
43+
tx-size-bytes-distribution:
44+
distribution: log-normal
45+
mu: 6.833
46+
sigma: 1.127
47+
tx-sharded-percentage: 1.0
48+
tx-validation-cpu-time-ms: 1.5
49+
tx-max-size-bytes: 16384
50+
tx-start-time: 300
51+
tx-stop-time: 300
52+
53+
################################################################################
54+
# Ranking Block Configuration
55+
################################################################################
56+
57+
# 1/leios-stage-length-slots, targeting one RB per pipeline.
58+
# Also 20s is current rate of praos blocks.
59+
rb-generation-probability: 5.0e-2
60+
# Eng. team targets 1kB as worst case upper bound.
61+
# Actual size fairly close.
62+
rb-head-size-bytes: 1024
63+
rb-body-max-size-bytes: 90112
64+
# Note: certificate generation/validation is not included in the
65+
# timings here, see cert-* fields.
66+
rb-generation-cpu-time-ms: 1.0
67+
rb-head-validation-cpu-time-ms: 1.0
68+
69+
# On average, no Txs directly embedded in blocks.
70+
rb-body-legacy-praos-payload-avg-size-bytes: 0
71+
rb-body-legacy-praos-payload-validation-cpu-time-ms-constant: 50.0
72+
# the -per-byte component is meant to be using size as a (bad)
73+
# proxy for the complexity of the Txs included.
74+
rb-body-legacy-praos-payload-validation-cpu-time-ms-per-byte: 0.0005
75+
76+
################################################################################
77+
# Input Block Configuration
78+
################################################################################
79+
80+
ib-generation-probability: 0.5
81+
ib-shards: 1
82+
ib-shard-period-length-slots: 10
83+
ib-shard-group-count: 1
84+
85+
# ProducerId 32
86+
# SlotNo 64
87+
# VRF proof 80
88+
# Body hash 32
89+
# RB Ref 32
90+
# Signature 64
91+
# Total 304
92+
#
93+
# NOTE: using a KES Signature (like for Praos headers)
94+
# would instead more than double the total to 668.
95+
# And even 828 including Op Cert.
96+
ib-head-size-bytes: 304
97+
# 98KB to optimize for 3 TCP round trips
98+
ib-body-avg-size-bytes: 196608
99+
ib-body-max-size-bytes: 196608
100+
# Here we also use praos blocks as ballpark estimate.
101+
# Sec 2.3 Forging, of the benchmark cluster report, lists
102+
# * Slot start to announced: 0.12975s
103+
ib-generation-cpu-time-ms: 130.0
104+
ib-head-validation-cpu-time-ms: 1.0
105+
ib-body-validation-cpu-time-ms-constant: 50.0
106+
ib-body-validation-cpu-time-ms-per-byte: 0.0005
107+
ib-diffusion-strategy: "freshest-first"
108+
109+
# Haskell prototype relay mini-protocol parameters.
110+
ib-diffusion-max-bodies-to-request: 1
111+
ib-diffusion-max-headers-to-request: 100
112+
ib-diffusion-max-window-size: 100
113+
114+
################################################################################
115+
# Endorsement Block Configuration
116+
################################################################################
117+
118+
# We want one per pipeline, but not too many.
119+
eb-generation-probability: 2.5
120+
# ProducerId 32
121+
# SlotNo 64
122+
# VRF proof 80
123+
# Signature 64
124+
# Total 240
125+
#
126+
# See Note about signatures on ib-head-size-bytes.
127+
eb-size-bytes-constant: 240
128+
# IB hash
129+
eb-size-bytes-per-ib: 32
130+
# Collecting the IBs to reference and cryptography are the main tasks.
131+
# A comparable task is maybe mempool snapshotting.
132+
# Sec 2.3 Forging, of the benchmark cluster report, lists
133+
# * Mempool snapshotting: 0.07252s
134+
# 75ms then seems a generous estimate for eb generation.
135+
eb-generation-cpu-time-ms: 75.0
136+
# Validating signature and vrf proof, as in other headers.
137+
eb-validation-cpu-time-ms: 1.0
138+
139+
eb-diffusion-strategy: "peer-order"
140+
141+
# Haskell prototype relay mini-protocol parameters.
142+
eb-diffusion-max-bodies-to-request: 1
143+
eb-diffusion-max-headers-to-request: 100
144+
eb-diffusion-max-window-size: 100
145+
146+
# The maximum age of EBs included in RBs.
147+
# A an EB from slot `s` can only be included in RBs
148+
# up to slot `s+eb-max-age-slots`.
149+
# In short leios we expect votes to diffuse within 3 stages lengths of
150+
# EB generation, we allow for 2 more stage lengths to account for
151+
# variance in the interval within RBs.
152+
eb-max-age-slots: 240
153+
154+
# The maximum age of EBs to be relayed.
155+
# An EB from slot `s` will only be relayed
156+
# up to slot `s+eb-max-age-for-relay-slots`.
157+
eb-max-age-for-relay-slots: 40
158+
159+
# The maximum size of transactions (in bytes) which an EB can reference.
160+
# Only relevant when running with the "full-without-ibs" variant.
161+
eb-referenced-txs-max-size-bytes: 16384000
162+
163+
################################################################################
164+
# Vote Configuration
165+
################################################################################
166+
167+
# Cryptography related values taken from [vote-spec](crypto-benchmarks.rs/Specification.md)
168+
# using weighted averages of 80% persistent and 20% non-persistent.
169+
170+
# vote-spec#Committe and quorum size
171+
#
172+
# Note: this is used as the expected amount of total weight of
173+
# generated votes in the sims.
174+
vote-generation-probability: 500.0
175+
# vote-spec#"Committe and quorum size"
176+
# 60% of `vote-generation-probability`
177+
vote-threshold: 300
178+
# vote-spec#"Generate vote" 0.8*135e-3 + 0.2*280e-3
179+
vote-generation-cpu-time-ms-constant: 164.0e-3
180+
# No benchmark yet.
181+
vote-generation-cpu-time-ms-per-ib: 0
182+
# vote-spec#"Verify vote" 0.8*670e-3 + 0.2*1.4
183+
vote-validation-cpu-time-ms: 816.0e-3
184+
# The `Vote` structure counted in the -per-eb already identifies slot
185+
# (in Eid) and voter. We can assume a vote bundle is all for the same
186+
# voter and slot, so for non-persistent voters we could factor their
187+
# PoolKeyHash (28bytes) here, but that is for 20% of cases.
188+
# More relevant if EB generation is very high.
189+
vote-bundle-size-bytes-constant: 0
190+
# vote-spec#Votes 0.8*90 + 0.2*164
191+
vote-bundle-size-bytes-per-eb: 105
192+
193+
vote-diffusion-strategy: "peer-order"
194+
195+
# Haskell prototype relay mini-protocol parameters.
196+
vote-diffusion-max-bodies-to-request: 1
197+
vote-diffusion-max-headers-to-request: 100
198+
vote-diffusion-max-window-size: 100
199+
200+
################################################################################
201+
# Certificate Configuration
202+
################################################################################
203+
204+
# vote-spec - certificate size plot.
205+
# Realistic stake distributions need about 7 kilobytes for the certificate.
206+
cert-size-bytes-constant: 7168
207+
cert-size-bytes-per-node: 0
208+
209+
# For certificate timings we have bulk figures for realistic scenarios,
210+
# so we do not attempt to give -per-node (i.e. per-voter) timings.
211+
#
212+
# vote-spec#"Generate certificate"
213+
cert-generation-cpu-time-ms-constant: 90.0
214+
cert-generation-cpu-time-ms-per-node: 0
215+
# vote-spec#"Verify certificate"
216+
cert-validation-cpu-time-ms-constant: 130.0
217+
cert-validation-cpu-time-ms-per-node: 0
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../topology-v2.yaml
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#!/usr/bin/env bash
2+
3+
set -e
4+
5+
cd "$(dirname "${BASH_SOURCE[0]}")"
6+
7+
mkfifo sim.log
8+
9+
grep -E -v '(Slot|No.*Generated|CpuTask|Sent|Lottery)' sim.log | xz -9c > sim.log.xz &
10+
11+
../../ols sim leios --leios-config-file config.yaml --topology-file network.yaml --shared-log-format JSON --conformance-events --output-file sim.log --output-seconds 300 > stdout 2> stderr
12+
13+
wait
14+
rm sim.log
15+
16+
cat << EOI > case.csv
17+
simulator,tps
18+
rust,$(basename $PWD)
19+
EOI
20+
21+
xzcat sim.log.xz \
22+
| ../../leios-trace-processor \
23+
+RTS -N5 -RTS \
24+
--trace-file /dev/stdin \
25+
--lifecycle-file lifecycle.csv \
26+
--cpu-file cpus.csv \
27+
--resource-file resources.csv \
28+
--receipt-file receipts.csv
29+
30+
pigz -9vf {lifecycle,cpus,resources,receipts}.csv
31+
32+
cat case.csv
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Simulator,IBPS
2+
Haskell,1.0

0 commit comments

Comments
 (0)