Skip to content

Commit e36422d

Browse files
bwbushBrian W Bush
bwbush
and
Brian W Bush
authored
Experiment to detect regressions in sim-cli (#505)
* Designed experiment to detect regressions in `sim-cli` * Created analysis notebook for detecting regressions * Analyzed experiment * Updated logbook --------- Co-authored-by: Brian W Bush <[email protected]>
1 parent 2aa40f6 commit e36422d

File tree

18 files changed

+2977
-0
lines changed

18 files changed

+2977
-0
lines changed

‎Logbook.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# Leios logbook
22

3+
## 2025-08-22
4+
5+
### Analysis for regressions in Rust simulator
6+
7+
The experiment [analysis/sims/regression/](analysis/sims/regression/) compares the behavior of all of the tagged versions of the Rust simulator, `sim-cli`, against each other when the same network topology and configuration file is used. This speeds detection of changes in behavior of the simulator.
8+
39
## 2025-08-21
410

511
### Analysis of diffusion of empty blocks on Cardano mainnet

‎analysis/sims/regression/analysis.ipynb‎

Lines changed: 2512 additions & 0 deletions
Large diffs are not rendered by default.

‎analysis/sims/regression/combine-results.sh‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#!/usr/bin/env nix-shell
2+
#!nix-shell -i bash -p gnused gzip pigz "rWrapper.override { packages = with rPackages; [ data_table R_utils bit64 ggplot2 magrittr stringr ]; }"
3+
4+
set -e
5+
6+
mkdir -p results/$d
7+
for f in lifecycle resources receipts cpus sizes
8+
do
9+
DIR=$(find experiment -type f -name $f.csv.gz \( -not -empty \) -printf %h\\n -quit)
10+
HL=$(sed -n -e '1p' "$DIR/case.csv")
11+
HR=$(zcat "$DIR/$f.csv.gz" | sed -n -e '1p')
12+
if [[ "$f" == "lifecycle" || "$f" == "resources" || "$f" == "sizes" ]]
13+
then
14+
FRACT=1.00
15+
else
16+
FRACT=0.50
17+
fi
18+
(
19+
echo "$HL,$HR"
20+
for g in $(find experiment -type f -name $f.csv.gz \( -not -empty \) -printf %h\\n)
21+
do
22+
if [ ! -e "$g/stderr" ]
23+
then
24+
echo "Skipping $g because it has no stderr." >> /dev/stderr
25+
elif [ -s "$g/stderr" ]
26+
then
27+
echo "Skipping $g because its stderr is not empty." >> /dev/stderr
28+
else
29+
BL=$(sed -n -e '2p' "$g/case.csv")
30+
zcat "$g/$f.csv.gz" | gawk 'FNR > 1 && rand() <= '"$FRACT"' { print "'"$BL"'" "," $0}'
31+
fi
32+
done
33+
) | pigz -p 3 -9c > results/$f.csv.gz
34+
R --vanilla << EOI > /dev/null
35+
require(data.table)
36+
sampleSize <- $FRACT
37+
print(sampleSize)
38+
$f <- fread("results/$f.csv.gz", stringsAsFactors=TRUE)
39+
save($f, sampleSize, file="results/$f.Rdata", compression_level=9)
40+
EOI
41+
done

‎analysis/sims/regression/experiment.list‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
experiment/0.1.0/run.sh
2+
experiment/1.0.0/run.sh
3+
experiment/1.1.0/run.sh
4+
experiment/1.2.0/run.sh

‎analysis/sims/regression/experiment/0.1.0/case.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
sim-cli,Tx start [s],Tx stop [s],Sim stop [s]
2+
0.1.0,60,660,1200

‎analysis/sims/regression/experiment/0.1.0/run.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../run.sh

‎analysis/sims/regression/experiment/0.1.0/summary.txt‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
INFO praos: sim_cli::events: 60001 transactions(s) were generated in total.
2+
INFO praos: sim_cli::events: 51 naive praos block(s) were published.
3+
INFO praos: sim_cli::events: 1149 slot(s) had no naive praos blocks.
4+
INFO praos: sim_cli::events: 60001 transaction(s) (90.00 MB) finalized in a naive praos block.
5+
INFO praos: sim_cli::events: 0 transaction(s) (0 B) did not reach a naive praos block.
6+
INFO praos: sim_cli::events: Pool 3 published 1 naive praos block(s)
7+
INFO praos: sim_cli::events: Pool 35 published 1 naive praos block(s)
8+
INFO praos: sim_cli::events: Pool 38 published 1 naive praos block(s)
9+
INFO praos: sim_cli::events: Pool 40 published 1 naive praos block(s)
10+
INFO praos: sim_cli::events: Pool 41 published 1 naive praos block(s)
11+
INFO praos: sim_cli::events: Pool 46 published 1 naive praos block(s)
12+
INFO praos: sim_cli::events: Pool 53 published 1 naive praos block(s)
13+
INFO praos: sim_cli::events: Pool 54 published 1 naive praos block(s)
14+
INFO praos: sim_cli::events: Pool 55 published 1 naive praos block(s)
15+
INFO praos: sim_cli::events: Pool 58 published 1 naive praos block(s)
16+
INFO praos: sim_cli::events: Pool 59 published 2 naive praos block(s)
17+
INFO praos: sim_cli::events: Pool 62 published 1 naive praos block(s)
18+
INFO praos: sim_cli::events: Pool 67 published 1 naive praos block(s)
19+
INFO praos: sim_cli::events: Pool 68 published 0 naive praos block(s)
20+
INFO praos: sim_cli::events: Pool 68 failed to publish 1 naive praos block(s) due to slot battles.
21+
INFO praos: sim_cli::events: Pool 69 published 1 naive praos block(s)
22+
INFO praos: sim_cli::events: Pool 73 published 1 naive praos block(s)
23+
INFO praos: sim_cli::events: Pool 86 published 1 naive praos block(s)
24+
INFO praos: sim_cli::events: Pool 93 published 1 naive praos block(s)
25+
INFO praos: sim_cli::events: Pool 109 published 1 naive praos block(s)
26+
INFO praos: sim_cli::events: Pool 121 published 1 naive praos block(s)
27+
INFO praos: sim_cli::events: Pool 130 published 1 naive praos block(s)
28+
INFO praos: sim_cli::events: Pool 132 published 1 naive praos block(s)
29+
INFO praos: sim_cli::events: Pool 136 published 1 naive praos block(s)
30+
INFO praos: sim_cli::events: Pool 351 published 1 naive praos block(s)
31+
INFO praos: sim_cli::events: Pool 362 published 1 naive praos block(s)
32+
INFO praos: sim_cli::events: Pool 429 published 1 naive praos block(s)
33+
INFO praos: sim_cli::events: Pool 431 published 2 naive praos block(s)
34+
INFO praos: sim_cli::events: Pool 477 published 2 naive praos block(s)
35+
INFO praos: sim_cli::events: Pool 478 published 1 naive praos block(s)
36+
INFO praos: sim_cli::events: Pool 517 published 1 naive praos block(s)
37+
INFO praos: sim_cli::events: Pool 519 published 1 naive praos block(s)
38+
INFO praos: sim_cli::events: Pool 521 published 1 naive praos block(s)
39+
INFO praos: sim_cli::events: Pool 528 published 1 naive praos block(s)
40+
INFO praos: sim_cli::events: Pool 534 published 1 naive praos block(s)
41+
INFO praos: sim_cli::events: Pool 535 published 2 naive praos block(s)
42+
INFO praos: sim_cli::events: Pool 539 published 1 naive praos block(s)
43+
INFO praos: sim_cli::events: Pool 540 published 1 naive praos block(s)
44+
INFO praos: sim_cli::events: Pool 542 published 1 naive praos block(s)
45+
INFO praos: sim_cli::events: Pool 544 published 1 naive praos block(s)
46+
INFO praos: sim_cli::events: Pool 560 published 1 naive praos block(s)
47+
INFO praos: sim_cli::events: Pool 562 published 1 naive praos block(s)
48+
INFO praos: sim_cli::events: Pool 739 published 1 naive praos block(s)
49+
INFO praos: sim_cli::events: Pool 741 published 1 naive praos block(s)
50+
INFO praos: sim_cli::events: Pool 742 published 1 naive praos block(s)
51+
INFO praos: sim_cli::events: Pool 744 published 2 naive praos block(s)
52+
INFO praos: sim_cli::events: Pool 745 published 1 naive praos block(s)
53+
INFO praos: sim_cli::events: Pool 747 published 1 naive praos block(s)
54+
INFO leios: sim_cli::events: 0 IB(s) were generated, on average 0.000 IB(s) per slot.
55+
INFO leios: sim_cli::events: 0 out of 60001 transaction(s) were included in at least one IB.
56+
INFO leios: sim_cli::events: The average age of the pending transactions is NaNs (stddev NaN).
57+
INFO leios: sim_cli::events: Each transaction was included in an average of NaN IB(s) (stddev NaN).
58+
INFO leios: sim_cli::events: Each IB contained an average of NaN transaction(s) (stddev NaN) and an average of 0 B (stddev 0 B). 0 IB(s) were empty.
59+
INFO leios: sim_cli::events: Each node received an average of 0.000 IB(s) (stddev 0.000).
60+
INFO leios: sim_cli::events: 52 EB(s) were generated; on average there were 0.043 EB(s) per slot.
61+
INFO leios: sim_cli::events: Each EB contained an average of 3231.346 transaction(s) (stddev 2908.723). 20 EB(s) were empty.
62+
INFO leios: sim_cli::events: Each EB contained an average of 0.000 IB(s) (stddev 0.000). 20 EB(s) were empty.
63+
INFO leios: sim_cli::events: Each IB was included in an average of NaN EB(s) (stddev NaN).
64+
INFO leios: sim_cli::events: 0 out of 0 IBs were included in at least one EB.
65+
INFO leios: sim_cli::events: 0 out of 0 IBs expired before they reached an EB.
66+
INFO leios: sim_cli::events: 22 out of 52 EBs expired before an EB from their stage reached an RB.
67+
INFO leios: sim_cli::events: 59941 out of 60001 transaction(s) were included in at least one EB.
68+
INFO leios: sim_cli::events: 25445 total votes were generated.
69+
INFO leios: sim_cli::events: Each stake pool produced an average of 117.801 vote(s) (stddev 23.533).
70+
INFO leios: sim_cli::events: Each EB received an average of 508.900 vote(s) (stddev 210.760).
71+
INFO leios: sim_cli::events: There were 8494 bundle(s) of votes. Each bundle contained 2.996 vote(s) (stddev 1.623).
72+
INFO leios: sim_cli::events: 30 L1 block(s) had a Leios endorsement.
73+
INFO leios: sim_cli::events: 59101 tx(s) (88.65 MB) were referenced by a Leios endorsement.
74+
INFO leios: sim_cli::events: 900 tx(s) (1.35 MB) were included directly in a Praos block.
75+
INFO leios: sim_cli::events: Spatial efficiency: 88.65 MB/5.63 MB (1574.784%) of Leios bytes were unique transactions.
76+
INFO leios: sim_cli::events: 31125 tx(s) (34.497%) referenced by a Leios endorsement were redundant.
77+
INFO leios: sim_cli::events: Each transaction took an average of NaNs (stddev NaN) to be included in an IB.
78+
INFO leios: sim_cli::events: Each transaction took an average of 21.158s (stddev 18.513) to be included in an EB.
79+
INFO leios: sim_cli::events: Each transaction took an average of 55.637s (stddev 26.138) to be included in a block.
80+
INFO network: sim_cli::events: 44940749 TX message(s) were sent. 44940749 of them were received (100.000%).
81+
INFO network: sim_cli::events: 0 IB message(s) were sent. 0 of them were received (NaN%).
82+
INFO network: sim_cli::events: 38492 EB message(s) were sent. 38492 of them were received (100.000%).
83+
INFO network: sim_cli::events: 6362006 Vote message(s) were sent. 6362006 of them were received (100.000%).

‎analysis/sims/regression/experiment/1.0.0/case.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
sim-cli,Tx start [s],Tx stop [s],Sim stop [s]
2+
1.0.0,60,660,1200

‎analysis/sims/regression/experiment/1.0.0/run.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../run.sh

‎analysis/sims/regression/experiment/1.0.0/summary.txt‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
INFO praos: sim_cli::events: 60001 transactions(s) were generated in total.
2+
INFO praos: sim_cli::events: 51 naive praos block(s) were published.
3+
INFO praos: sim_cli::events: 1149 slot(s) had no naive praos blocks.
4+
INFO praos: sim_cli::events: 60001 transaction(s) (90.00 MB) finalized in a naive praos block.
5+
INFO praos: sim_cli::events: 0 transaction(s) (0 B) did not reach a naive praos block.
6+
INFO praos: sim_cli::events: Pool 3 published 1 naive praos block(s)
7+
INFO praos: sim_cli::events: Pool 35 published 1 naive praos block(s)
8+
INFO praos: sim_cli::events: Pool 38 published 1 naive praos block(s)
9+
INFO praos: sim_cli::events: Pool 40 published 1 naive praos block(s)
10+
INFO praos: sim_cli::events: Pool 41 published 1 naive praos block(s)
11+
INFO praos: sim_cli::events: Pool 46 published 1 naive praos block(s)
12+
INFO praos: sim_cli::events: Pool 53 published 1 naive praos block(s)
13+
INFO praos: sim_cli::events: Pool 54 published 1 naive praos block(s)
14+
INFO praos: sim_cli::events: Pool 55 published 1 naive praos block(s)
15+
INFO praos: sim_cli::events: Pool 58 published 1 naive praos block(s)
16+
INFO praos: sim_cli::events: Pool 59 published 2 naive praos block(s)
17+
INFO praos: sim_cli::events: Pool 62 published 1 naive praos block(s)
18+
INFO praos: sim_cli::events: Pool 67 published 1 naive praos block(s)
19+
INFO praos: sim_cli::events: Pool 68 published 0 naive praos block(s)
20+
INFO praos: sim_cli::events: Pool 68 failed to publish 1 naive praos block(s) due to slot battles.
21+
INFO praos: sim_cli::events: Pool 69 published 1 naive praos block(s)
22+
INFO praos: sim_cli::events: Pool 73 published 1 naive praos block(s)
23+
INFO praos: sim_cli::events: Pool 86 published 1 naive praos block(s)
24+
INFO praos: sim_cli::events: Pool 93 published 1 naive praos block(s)
25+
INFO praos: sim_cli::events: Pool 109 published 1 naive praos block(s)
26+
INFO praos: sim_cli::events: Pool 121 published 1 naive praos block(s)
27+
INFO praos: sim_cli::events: Pool 130 published 1 naive praos block(s)
28+
INFO praos: sim_cli::events: Pool 132 published 1 naive praos block(s)
29+
INFO praos: sim_cli::events: Pool 136 published 1 naive praos block(s)
30+
INFO praos: sim_cli::events: Pool 351 published 1 naive praos block(s)
31+
INFO praos: sim_cli::events: Pool 362 published 1 naive praos block(s)
32+
INFO praos: sim_cli::events: Pool 429 published 1 naive praos block(s)
33+
INFO praos: sim_cli::events: Pool 431 published 2 naive praos block(s)
34+
INFO praos: sim_cli::events: Pool 477 published 2 naive praos block(s)
35+
INFO praos: sim_cli::events: Pool 478 published 1 naive praos block(s)
36+
INFO praos: sim_cli::events: Pool 517 published 1 naive praos block(s)
37+
INFO praos: sim_cli::events: Pool 519 published 1 naive praos block(s)
38+
INFO praos: sim_cli::events: Pool 521 published 1 naive praos block(s)
39+
INFO praos: sim_cli::events: Pool 528 published 1 naive praos block(s)
40+
INFO praos: sim_cli::events: Pool 534 published 1 naive praos block(s)
41+
INFO praos: sim_cli::events: Pool 535 published 2 naive praos block(s)
42+
INFO praos: sim_cli::events: Pool 539 published 1 naive praos block(s)
43+
INFO praos: sim_cli::events: Pool 540 published 1 naive praos block(s)
44+
INFO praos: sim_cli::events: Pool 542 published 1 naive praos block(s)
45+
INFO praos: sim_cli::events: Pool 544 published 1 naive praos block(s)
46+
INFO praos: sim_cli::events: Pool 560 published 1 naive praos block(s)
47+
INFO praos: sim_cli::events: Pool 562 published 1 naive praos block(s)
48+
INFO praos: sim_cli::events: Pool 739 published 1 naive praos block(s)
49+
INFO praos: sim_cli::events: Pool 741 published 1 naive praos block(s)
50+
INFO praos: sim_cli::events: Pool 742 published 1 naive praos block(s)
51+
INFO praos: sim_cli::events: Pool 744 published 2 naive praos block(s)
52+
INFO praos: sim_cli::events: Pool 745 published 1 naive praos block(s)
53+
INFO praos: sim_cli::events: Pool 747 published 1 naive praos block(s)
54+
INFO leios: sim_cli::events: 0 IB(s) were generated, on average 0.000 IB(s) per slot.
55+
INFO leios: sim_cli::events: 0 out of 60001 transaction(s) were included in at least one IB.
56+
INFO leios: sim_cli::events: The average age of the pending transactions is NaNs (stddev NaN).
57+
INFO leios: sim_cli::events: Each transaction was included in an average of NaN IB(s) (stddev NaN).
58+
INFO leios: sim_cli::events: Each IB contained an average of NaN transaction(s) (stddev NaN) and an average of 0 B (stddev 0 B). 0 IB(s) were empty.
59+
INFO leios: sim_cli::events: Each node received an average of 0.000 IB(s) (stddev 0.000).
60+
INFO leios: sim_cli::events: 52 EB(s) were generated; on average there were 0.043 EB(s) per slot.
61+
INFO leios: sim_cli::events: Each EB contained an average of 2635.731 transaction(s) (stddev 2703.129). 20 EB(s) were empty.
62+
INFO leios: sim_cli::events: Each EB contained an average of 0.000 IB(s) (stddev 0.000). 20 EB(s) were empty.
63+
INFO leios: sim_cli::events: Each IB was included in an average of NaN EB(s) (stddev NaN).
64+
INFO leios: sim_cli::events: 0 out of 0 IBs were included in at least one EB.
65+
INFO leios: sim_cli::events: 0 out of 0 IBs expired before they reached an EB.
66+
INFO leios: sim_cli::events: 12 out of 52 EBs expired before an EB from their stage reached an RB.
67+
INFO leios: sim_cli::events: 59941 out of 60001 transaction(s) were included in at least one EB.
68+
INFO leios: sim_cli::events: 25444 total votes were generated.
69+
INFO leios: sim_cli::events: Each stake pool produced an average of 117.796 vote(s) (stddev 23.370).
70+
INFO leios: sim_cli::events: Each EB received an average of 508.880 vote(s) (stddev 210.910).
71+
INFO leios: sim_cli::events: There were 8497 bundle(s) of votes. Each bundle contained 2.994 vote(s) (stddev 1.624).
72+
INFO leios: sim_cli::events: 40 L1 block(s) had a Leios endorsement.
73+
INFO leios: sim_cli::events: 59461 tx(s) (89.19 MB) were referenced by a Leios endorsement.
74+
INFO leios: sim_cli::events: 540 tx(s) (810 kB) were included directly in a Praos block.
75+
INFO leios: sim_cli::events: Spatial efficiency: 89.19 MB/4.72 MB (1890.317%) of Leios bytes were unique transactions.
76+
INFO leios: sim_cli::events: 39339 tx(s) (39.817%) referenced by a Leios endorsement were redundant.
77+
INFO leios: sim_cli::events: Each transaction took an average of NaNs (stddev NaN) to be included in an IB.
78+
INFO leios: sim_cli::events: Each transaction took an average of 21.177s (stddev 18.506) to be included in an EB.
79+
INFO leios: sim_cli::events: Each transaction took an average of 48.868s (stddev 29.308) to be included in a block.
80+
INFO network: sim_cli::events: 44940749 TX message(s) were sent. 44940749 of them were received (100.000%).
81+
INFO network: sim_cli::events: 0 IB message(s) were sent. 0 of them were received (NaN%).
82+
INFO network: sim_cli::events: 38492 EB message(s) were sent. 38492 of them were received (100.000%).
83+
INFO network: sim_cli::events: 6364253 Vote message(s) were sent. 6364253 of them were received (100.000%).

0 commit comments

Comments
 (0)