Skip to content

Commit 8faa330

Browse files
bwbushbwbush
authored
Simulator analysis at tag leios-2025w13 (#273)
* Configurations for varying IB delivery * Configured additional experiments * Adjusted experimental design * Tweaked queries * Analysis of tag `leios-2025w13` * Updated logbook
1 parent 686ab3b commit 8faa330

File tree

10 files changed

+1878
-9
lines changed

10 files changed

+1878
-9
lines changed

‎Logbook.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,16 @@
2525
- The `close-and-random` peers chosen for each node are now properly listed as `producers`.
2626
- Variance in upstream peers, and hence access to blocks, is now much smaller.
2727

28+
### Analysis of simulations at tag `leios-2025w13`
29+
30+
The Haskell and Rust simulators, at tag [leios-2025w13](https://github.com/input-output-hk/ouroboros-leios/releases/tag/leios-2025w13), were used to simulation 198 scenarios of Short Leios for varied IB production rate, IB size, and network topology, CPU limits, and protocol flags. Comprehensive results are in the Jupyter notebook [analysis/sims/2025w13/analysis.ipynb](analysis/sims/2025w13/analysis.ipynb). New analyses focused on exploring simulator settings related to message delivery and CPU. The simulations resolved most of the prior outstanding issues that arose in comparing simulators, but they unearthed new ones.
31+
32+
1. Studied how limiting available CPU affects the propagation of messages like IBs, where they might potentially be lost if insufficient CPU is available. *CPU can impact diffusion of IBs under some stressful scenarios.*
33+
2. Compared freshest-first versus oldest-first vote propagation. *Freshest first (arguably) may improve reliability of IB delivery.*
34+
3. Compared an extended voting period versus a limited one in the Haskell simulation. *Extended voting makes little difference except for rare improvements in reliable delivery of votes.*
35+
36+
The qualitative discrepancies between the Haskell and Rust simulators' results are under investigation to determine whether they are *bona fide* differences due to the resolution of the simulators instead of infidelities in the simulations themselves.
37+
2838
### Tools
2939

3040
### Egress Traffic Calculator

‎analysis/sims/2025w13/analysis.ipynb‎

Lines changed: 1398 additions & 0 deletions
Large diffs are not rendered by default.

‎analysis/sims/2025w13/build-experiments.sh‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
set -ev
44

5+
56
LABEL=default
67
NETWORK=100-nodes
78
MAX_SLOT=600
@@ -22,3 +23,39 @@ do
2223
done
2324
done
2425
done
26+
27+
28+
LABEL=default
29+
IB_SIZE=98304
30+
EB_RATE=2.0
31+
STAGE_LENGTH=40
32+
MAX_SLOT=600
33+
34+
for SIMULATOR in haskell rust
35+
do
36+
for NETWORK in 100-nodes-4cpus 100-nodes-6cpus 100-nodes-8cpus
37+
do
38+
for IB_RATE in 1.0 5.0 25.0
39+
do
40+
./make-experiment.sh $SIMULATOR $MAX_SLOT $LABEL $NETWORK $IB_RATE $IB_SIZE $EB_RATE $STAGE_LENGTH
41+
done
42+
done
43+
done
44+
45+
46+
NETWORK=100-nodes
47+
IB_SIZE=98304
48+
EB_RATE=2.0
49+
STAGE_LENGTH=40
50+
MAX_SLOT=600
51+
52+
for SIMULATOR in haskell rust
53+
do
54+
for LABEL in default extended-voting oldest-first peer-order
55+
do
56+
for IB_RATE in 1.0 5.0 25.0
57+
do
58+
./make-experiment.sh $SIMULATOR $MAX_SLOT $LABEL $NETWORK $IB_RATE $IB_SIZE $EB_RATE $STAGE_LENGTH
59+
done
60+
done
61+
done

‎analysis/sims/2025w13/config.oldest-first.yaml‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# yaml-language-server: $schema=./config.schema.json
2+
3+
################################################################################
4+
# Simulation Configuration File
5+
################################################################################
6+
#
7+
# This file contains the default configuration for running Leios simulations in
8+
# the Haskell simulation (`simulation/`) and the Rust simulation (`sim-rs/`).
9+
#
10+
################################################################################
11+
# Simulation Configuration
12+
################################################################################
13+
14+
relay-strategy: "request-from-first"
15+
tcp-congestion-control: true
16+
multiplex-mini-protocols: true
17+
simulate-transactions: false
18+
treat-blocks-as-full: false
19+
cleanup-policies: ["cleanup-expired-vote"]
20+
21+
################################################################################
22+
# Leios Protocol Configuration
23+
################################################################################
24+
25+
leios-variant: short
26+
leios-stage-length-slots: 20
27+
leios-stage-active-voting-slots: 1
28+
leios-vote-send-recv-stages: false
29+
leios-header-diffusion-time-ms: 1000.0
30+
# TODO: revise default
31+
praos-chain-quality: 20
32+
33+
################################################################################
34+
# Transaction Configuration
35+
################################################################################
36+
37+
tx-generation-distribution:
38+
distribution: exp
39+
lambda: 0.85
40+
scale: 1000.0
41+
tx-size-bytes-distribution:
42+
distribution: log-normal
43+
mu: 6.833
44+
sigma: 1.127
45+
tx-validation-cpu-time-ms: 1.5
46+
tx-max-size-bytes: 16384
47+
48+
################################################################################
49+
# Ranking Block Configuration
50+
################################################################################
51+
52+
# 1/leios-stage-length-slots, targeting one RB per pipeline.
53+
# Also 20s is current rate of praos blocks.
54+
rb-generation-probability: 5.0e-2
55+
# Eng. team targets 1kB as worst case upper bound.
56+
# Actual size fairly close.
57+
rb-head-size-bytes: 1024
58+
rb-body-max-size-bytes: 90112
59+
# Note: certificate generation/validation is not included in the
60+
# timings here, see cert-* fields.
61+
rb-generation-cpu-time-ms: 1.0
62+
rb-head-validation-cpu-time-ms: 1.0
63+
64+
# On average, no Txs directly embedded in blocks.
65+
rb-body-legacy-praos-payload-avg-size-bytes: 0
66+
rb-body-legacy-praos-payload-validation-cpu-time-ms-constant: 50.0
67+
# the -per-byte component is meant to be using size as a (bad)
68+
# proxy for the complexity of the Txs included.
69+
rb-body-legacy-praos-payload-validation-cpu-time-ms-per-byte: 0.0005
70+
71+
################################################################################
72+
# Input Block Configuration
73+
################################################################################
74+
75+
ib-generation-probability: 5.0
76+
ib-shards: 1
77+
78+
# ProducerId 32
79+
# SlotNo 64
80+
# VRF proof 80
81+
# Body hash 32
82+
# RB Ref 32
83+
# Signature 64
84+
# Total 304
85+
#
86+
# NOTE: using a KES Signature (like for Praos headers)
87+
# would instead more than double the total to 668.
88+
# And even 828 including Op Cert.
89+
ib-head-size-bytes: 304
90+
# 98KB to optimize for 3 TCP round trips
91+
ib-body-avg-size-bytes: 98304
92+
ib-body-max-size-bytes: 327680
93+
# Here we also use praos blocks as ballpark estimate.
94+
# Sec 2.3 Forging, of the benchmark cluster report, lists
95+
# * Slot start to announced: 0.12975s
96+
ib-generation-cpu-time-ms: 130.0
97+
ib-head-validation-cpu-time-ms: 1.0
98+
ib-body-validation-cpu-time-ms-constant: 50.0
99+
ib-body-validation-cpu-time-ms-per-byte: 0.0005
100+
ib-diffusion-strategy: "oldest-first"
101+
102+
# Haskell prototype relay mini-protocol parameters.
103+
ib-diffusion-max-bodies-to-request: 1
104+
ib-diffusion-max-headers-to-request: 100
105+
ib-diffusion-max-window-size: 100
106+
107+
################################################################################
108+
# Endorsement Block Configuration
109+
################################################################################
110+
111+
# We want one per pipeline, but not too many.
112+
eb-generation-probability: 1.5
113+
# ProducerId 32
114+
# SlotNo 64
115+
# VRF proof 80
116+
# Signature 64
117+
# Total 240
118+
#
119+
# See Note about signatures on ib-head-size-bytes.
120+
eb-size-bytes-constant: 240
121+
# IB hash
122+
eb-size-bytes-per-ib: 32
123+
# Collecting the IBs to reference and cryptography are the main tasks.
124+
# A comparable task is maybe mempool snapshotting.
125+
# Sec 2.3 Forging, of the benchmark cluster report, lists
126+
# * Mempool snapshotting: 0.07252s
127+
# 75ms then seems a generous estimate for eb generation.
128+
eb-generation-cpu-time-ms: 75.0
129+
# Validating signature and vrf proof, as in other headers.
130+
eb-validation-cpu-time-ms: 1.0
131+
132+
eb-diffusion-strategy: "peer-order"
133+
134+
# Haskell prototype relay mini-protocol parameters.
135+
eb-diffusion-max-bodies-to-request: 1
136+
eb-diffusion-max-headers-to-request: 100
137+
eb-diffusion-max-window-size: 100
138+
139+
# The maximum age of EBs included in RBs.
140+
# A an EB from slot `s` can only be included in RBs
141+
# up to slot `s+eb-max-age-slots`.
142+
# In short leios we expect votes to diffuse within 3 stages lengths of
143+
# EB generation, we allow for 2 more stage lengths to account for
144+
# variance in the interval within RBs.
145+
eb-max-age-slots: 100
146+
147+
# The maximum age of EBs to be relayed.
148+
# An EB from slot `s` will only be relayed
149+
# up to slot `s+eb-max-age-for-relay-slots`.
150+
eb-max-age-for-relay-slots: 40
151+
152+
################################################################################
153+
# Vote Configuration
154+
################################################################################
155+
156+
# Cryptography related values taken from [vote-spec](crypto-benchmarks.rs/Specification.md)
157+
# using weighted averages of 80% persistent and 20% non-persistent.
158+
159+
# vote-spec#Committe and quorum size
160+
#
161+
# Note: this is used as the expected amount of total weight of
162+
# generated votes in the sims.
163+
vote-generation-probability: 500.0
164+
# vote-spec#"Committe and quorum size"
165+
# 60% of `vote-generation-probability`
166+
vote-threshold: 300
167+
# vote-spec#"Generate vote" 0.8*135e-3 + 0.2*280e-3
168+
vote-generation-cpu-time-ms-constant: 164.0e-3
169+
# No benchmark yet.
170+
vote-generation-cpu-time-ms-per-ib: 0
171+
# vote-spec#"Verify vote" 0.8*670e-3 + 0.2*1.4
172+
vote-validation-cpu-time-ms: 816.0e-3
173+
# The `Vote` structure counted in the -per-eb already identifies slot
174+
# (in Eid) and voter. We can assume a vote bundle is all for the same
175+
# voter and slot, so for non-persistent voters we could factor their
176+
# PoolKeyHash (28bytes) here, but that is for 20% of cases.
177+
# More relevant if EB generation is very high.
178+
vote-bundle-size-bytes-constant: 0
179+
# vote-spec#Votes 0.8*90 + 0.2*164
180+
vote-bundle-size-bytes-per-eb: 105
181+
182+
vote-diffusion-strategy: "peer-order"
183+
184+
# Haskell prototype relay mini-protocol parameters.
185+
vote-diffusion-max-bodies-to-request: 1
186+
vote-diffusion-max-headers-to-request: 100
187+
vote-diffusion-max-window-size: 100
188+
189+
################################################################################
190+
# Certificate Configuration
191+
################################################################################
192+
193+
# vote-spec - certificate size plot.
194+
# Realistic stake distributions need about 7 kilobytes for the certificate.
195+
cert-size-bytes-constant: 7168
196+
cert-size-bytes-per-node: 0
197+
198+
# For certificate timings we have bulk figures for realistic scenarios,
199+
# so we do not attempt to give -per-node (i.e. per-voter) timings.
200+
#
201+
# vote-spec#"Generate certificate"
202+
cert-generation-cpu-time-ms-constant: 90.0
203+
cert-generation-cpu-time-ms-per-node: 0
204+
# vote-spec#"Verify certificate"
205+
cert-validation-cpu-time-ms-constant: 130.0
206+
cert-validation-cpu-time-ms-per-node: 0

0 commit comments

Comments
 (0)