Skip to content

Commit adec324

Browse files
jpraynaudjpraynaud
authored
Merge pull request #1718 from input-output-hk/jpraynaud/benchmark-aggregator-prover
Benchmark aggregator prover tool
2 parents b570301 + ce945e0 commit adec324

File tree

3 files changed

+406
-0
lines changed

3 files changed

+406
-0
lines changed

‎mithril-test-lab/benchmark/aggregator-prover/README.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Benchmark aggregator prover route
2+
3+
This tool will run a set of benchmarks (based on [Apache Benchmark](https://httpd.apache.org/docs/2.4/programs/ab.html)) on the aggregator route of an aggregator given a list of transactions. It will produce a CSV file with results of the benchmarks.
4+
5+
First set enviroment variables:
6+
```bash
7+
# Aggregator endpoint
8+
export AGGREGATOR_ENDPOINT=https://aggregator.testing-mainnet.api.mithril.network/aggregator
9+
10+
# Transactions file to prove
11+
export TRANSACTIONS_FILE=transactions-mainnet.txt
12+
13+
# Transactions proved per request range definition
14+
export TRANSACTIONS_PER_REQUEST_MIN=1
15+
export TRANSACTIONS_PER_REQUEST_MAX=3
16+
export TRANSACTIONS_PER_REQUEST_STEP=1
17+
18+
# Apache benchmark total request sent per benchmark
19+
export AB_TOTAL_REQUESTS=1000
20+
21+
# Apache benchmark concurrency level range definition
22+
export AB_CONCURRENCY_MIN=50
23+
export AB_CONCURRENCY_MAX=100
24+
export AB_CONCURRENCY_STEP=50
25+
```
26+
27+
Then, run the benchmarks:
28+
```bash
29+
./benchmark-aggregator-prover.sh
30+
```
31+
32+
Which will output these type of results:
33+
```bash
34+
MITHRIL AGGREGATOR PROVER ROUTE BENCHMARK
35+
36+
>> Aggregator endpoint: https://aggregator.testing-mainnet.api.mithril.network/aggregator
37+
>> Aggregator route: /proof/cardano-transaction
38+
>> Transactions file: transactions-mainnet.txt
39+
>> Transactions available: [100]
40+
>> Transactions per request range: [1 2 3]
41+
>> AB concurrency range: [50 100]
42+
>> AB total requests per run: [1000]
43+
>> AB total runs: 6
44+
>> Output file: benchmark.csv
45+
46+
>> [#1/6] Running stress test with 1000 requests with 1 transactions per request and 50 concurrency
47+
Completed 100 requests
48+
Completed 200 requests
49+
Completed 300 requests
50+
Completed 400 requests
51+
Completed 500 requests
52+
Completed 600 requests
53+
Completed 700 requests
54+
Completed 800 requests
55+
Completed 900 requests
56+
Completed 1000 requests
57+
Finished 1000 requests
58+
>>>> Success (83.23 requests/s)
59+
60+
>> [#2/6] Running stress test with 1000 requests with 1 transactions per request and 100 concurrency
61+
Completed 100 requests
62+
Completed 200 requests
63+
Completed 300 requests
64+
Completed 400 requests
65+
Completed 500 requests
66+
Completed 600 requests
67+
Completed 700 requests
68+
Completed 800 requests
69+
Completed 900 requests
70+
Completed 1000 requests
71+
Finished 1000 requests
72+
>>>> Success (80.68 requests/s)
73+
74+
>> [#3/6] Running stress test with 1000 requests with 2 transactions per request and 50 concurrency
75+
Completed 100 requests
76+
Completed 200 requests
77+
Completed 300 requests
78+
Completed 400 requests
79+
Completed 500 requests
80+
Completed 600 requests
81+
Completed 700 requests
82+
Completed 800 requests
83+
Completed 900 requests
84+
Completed 1000 requests
85+
Finished 1000 requests
86+
>>>> Success (57.50 requests/s)
87+
88+
>> [#4/6] Running stress test with 1000 requests with 2 transactions per request and 100 concurrency
89+
Completed 100 requests
90+
Completed 200 requests
91+
Completed 300 requests
92+
Completed 400 requests
93+
Completed 500 requests
94+
Completed 600 requests
95+
Completed 700 requests
96+
Completed 800 requests
97+
Completed 900 requests
98+
Completed 1000 requests
99+
Finished 1000 requests
100+
>>>> Success (60.61 requests/s)
101+
102+
>> [#5/6] Running stress test with 1000 requests with 3 transactions per request and 50 concurrency
103+
Completed 100 requests
104+
Completed 200 requests
105+
Completed 300 requests
106+
Completed 400 requests
107+
Completed 500 requests
108+
Completed 600 requests
109+
Completed 700 requests
110+
Completed 800 requests
111+
Completed 900 requests
112+
Completed 1000 requests
113+
Finished 1000 requests
114+
>>>> Success (42.86 requests/s)
115+
116+
>> [#6/6] Running stress test with 1000 requests with 3 transactions per request and 100 concurrency
117+
Completed 100 requests
118+
Completed 200 requests
119+
Completed 300 requests
120+
Completed 400 requests
121+
Completed 500 requests
122+
Completed 600 requests
123+
Completed 700 requests
124+
Completed 800 requests
125+
Completed 900 requests
126+
Completed 1000 requests
127+
Finished 1000 requests
128+
>>>> Success (45.60 requests/s)
129+
130+
>> Benchmark completed:
131+
132+
total_requests,concurrency,transactions/request,requests/s
133+
1000,50,1,83.23
134+
1000,100,1,80.68
135+
1000,50,2,57.50
136+
1000,100,2,60.61
137+
1000,50,3,42.86
138+
1000,100,3,45.60
139+
140+
```

‎mithril-test-lab/benchmark/aggregator-prover/benchmark-aggregator-prover.sh‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
#!/bin/bash
2+
3+
set -e
4+
5+
check_requirements() {
6+
which ab >/dev/null ||
7+
error "It seems 'ab' is not installed or not in the path.";
8+
9+
}
10+
11+
error() {
12+
echo
13+
echo "ERROR: $1";
14+
exit 1;
15+
}
16+
17+
display_help() {
18+
echo
19+
if [ -n "${1-""}" ]; then echo "ERROR: ${1}"; echo; fi
20+
echo "HELP"
21+
echo "$0"
22+
echo
23+
echo "Benchmark a Mithril aggregator prover route"
24+
echo
25+
echo "Usage: $0"
26+
echo
27+
echo "Available configuration environment variables:"
28+
echo "- DEBUG: activate debug output"
29+
echo "- AGGREGATOR_ENDPOINT: the aggregator endpoint"
30+
echo "- TRANSACTIONS_FILE: the file containing the transactions hashes"
31+
echo "- TRANSACTIONS_PER_REQUEST_MIN: the minimum number of transactions per request"
32+
echo "- TRANSACTIONS_PER_REQUEST_MAX: the maximum number of transactions per request"
33+
echo "- TRANSACTIONS_PER_REQUEST_STEP: the step between each number of transactions per request"
34+
echo "- AB_TOTAL_REQUESTS: the total number of requests"
35+
echo "- AB_CONCURRENCY_MIN: the minimum concurrency level"
36+
echo "- AB_CONCURRENCY_MAX: the maximum concurrency level"
37+
echo "- AB_CONCURRENCY_STEP: the step between each concurrency level"
38+
echo
39+
exit 1;
40+
}
41+
42+
echo ""
43+
echo "MITHRIL AGGREGATOR PROVER ROUTE BENCHMARK"
44+
45+
check_requirements
46+
47+
# Debug mode
48+
if [ -v DEBUG ]; then
49+
set -x
50+
fi
51+
52+
AGGREGATOR_PROVER_ROUTE="/proof/cardano-transaction"
53+
54+
# Check if all env vars are set
55+
if [ -z "${AGGREGATOR_ENDPOINT}" ]; then
56+
display_help "Missing environment variable: AGGREGATOR_ENDPOINT"
57+
fi
58+
59+
if [ -z "${TRANSACTIONS_FILE}" ]; then
60+
display_help "Missing environment variable: TRANSACTIONS_FILE"
61+
fi
62+
63+
if [ -z "${TRANSACTIONS_PER_REQUEST_MIN}" ]; then
64+
TRANSACTIONS_PER_REQUEST_MIN=0
65+
fi
66+
67+
if [ -z "${TRANSACTIONS_PER_REQUEST_MAX}" ]; then
68+
TRANSACTIONS_PER_REQUEST_MAX=100
69+
fi
70+
TRANSACTIONS=$(tr "\n" " " < "$TRANSACTIONS_FILE")
71+
TRANSACTIONS_AVAILABLE=$(echo "$TRANSACTIONS" | wc -w | xargs)
72+
TRANSACTIONS_PER_REQUEST_MAX=$(( TRANSACTIONS_AVAILABLE < TRANSACTIONS_PER_REQUEST_MAX ? TRANSACTIONS_AVAILABLE : TRANSACTIONS_PER_REQUEST_MAX ))
73+
74+
if [ -z "${TRANSACTIONS_PER_REQUEST_STEP}" ]; then
75+
TRANSACTIONS_PER_REQUEST_STEP=5
76+
fi
77+
78+
if [ -z "${AB_TOTAL_REQUESTS}" ]; then
79+
AB_TOTAL_REQUESTS=1000
80+
fi
81+
82+
if [ -z "${AB_CONCURRENCY_MIN}" ]; then
83+
AB_CONCURRENCY_MIN=0
84+
fi
85+
86+
if [ -z "${AB_CONCURRENCY_MAX}" ]; then
87+
AB_CONCURRENCY_MAX=100
88+
fi
89+
90+
if [ -z "${AB_CONCURRENCY_STEP}" ]; then
91+
AB_CONCURRENCY_STEP=10
92+
fi
93+
94+
if [ -z "${AB_TIMEOUT}" ]; then
95+
AB_TIMEOUT=180
96+
fi
97+
98+
if [ -z "${OUT_FILE}" ]; then
99+
OUT_FILE="benchmark.csv"
100+
rm -f $OUT_FILE
101+
fi
102+
mkdir -p "$(dirname "$OUT_FILE")"
103+
104+
# Run stress test
105+
RUN_STRESS_TEST() {
106+
AGGREGATOR_ENDPOINT=$1
107+
TRANSACTIONS_PER_REQUEST=$2
108+
TRANSACTIONS_PER_REQUEST=$(( TRANSACTIONS_PER_REQUEST > 0 ? TRANSACTIONS_PER_REQUEST : 1 ))
109+
AB_TOTAL_REQUESTS=$3
110+
AB_CONCURRENCY=$4
111+
AB_CONCURRENCY=$(( AB_CONCURRENCY > 0 ? AB_CONCURRENCY : 0 ))
112+
AB_TOTAL_REQUESTS=$(( AB_TOTAL_REQUESTS > AB_CONCURRENCY ? AB_TOTAL_REQUESTS : AB_CONCURRENCY ))
113+
OUT_FILE=$5
114+
INDEX_RUN=$6
115+
TOTAL_RUN=$7
116+
TRANSACTIONS_FILE=$8
117+
TMP_FILE="test.tmp"
118+
echo ">> [#$INDEX_RUN/$TOTAL_RUN] Running stress test with $AB_TOTAL_REQUESTS requests with $TRANSACTIONS_PER_REQUEST transactions per request and $AB_CONCURRENCY concurrency"
119+
TRANSACTIONS_LIST=$(head -n $TRANSACTIONS_PER_REQUEST "$TRANSACTIONS_FILE" | tr "\n" ",")
120+
AGGREGATOR_PROVER_URL="${AGGREGATOR_ENDPOINT}${AGGREGATOR_PROVER_ROUTE}?transaction_hashes=$TRANSACTIONS_LIST"
121+
if ab -n $AB_TOTAL_REQUESTS -c $AB_CONCURRENCY -s "$AB_TIMEOUT" "$AGGREGATOR_PROVER_URL" > $TMP_FILE ; then
122+
REQUESTS_PER_SECOND=$(cat $TMP_FILE | awk '/Requests per second:/ {print $4}')
123+
if [[ $INDEX_RUN -eq 1 ]] ; then
124+
echo "total_requests,concurrency,transactions/request,requests/s" >> "$OUT_FILE"
125+
fi
126+
echo "$AB_TOTAL_REQUESTS,$AB_CONCURRENCY,$TRANSACTIONS_PER_REQUEST,$REQUESTS_PER_SECOND" >> "$OUT_FILE"
127+
echo ">>>> Success ($REQUESTS_PER_SECOND requests/s)"
128+
else
129+
echo ">>>> Failure"
130+
exit
131+
fi
132+
rm -f $TMP_FILE
133+
echo ""
134+
}
135+
136+
# Run aggregator benchmark over a range of transactions and concurrency levels
137+
TRANSACTIONS_PER_REQUEST_RANGE=$(seq -s ' ' "$TRANSACTIONS_PER_REQUEST_MIN" "$TRANSACTIONS_PER_REQUEST_STEP" $TRANSACTIONS_PER_REQUEST_MAX)
138+
AB_CONCURRENCY_RANGE=$(seq -s ' ' "$AB_CONCURRENCY_MIN" "$AB_CONCURRENCY_STEP" "$AB_CONCURRENCY_MAX")
139+
TRANSACTIONS_PER_REQUEST_RANGE_LENGTH=$(( $(echo "$TRANSACTIONS_PER_REQUEST_RANGE" | grep -o " " | wc -l) + 1 ))
140+
AB_CONCURRENCY_RANGE_LENGTH=$(( $(echo "$AB_CONCURRENCY_RANGE" | grep -o " " | wc -l) + 1 ))
141+
TOTAL_RUN=$(( TRANSACTIONS_PER_REQUEST_RANGE_LENGTH * AB_CONCURRENCY_RANGE_LENGTH ))
142+
echo
143+
echo ">> Aggregator endpoint: $AGGREGATOR_ENDPOINT"
144+
echo ">> Aggregator route: $AGGREGATOR_PROVER_ROUTE"
145+
echo ">> Transactions file: $TRANSACTIONS_FILE"
146+
echo ">> Transactions available: [$TRANSACTIONS_AVAILABLE]"
147+
echo ">> Transactions per request range: [$TRANSACTIONS_PER_REQUEST_RANGE]"
148+
echo ">> AB concurrency range: [$AB_CONCURRENCY_RANGE]"
149+
echo ">> AB total requests per run: [$AB_TOTAL_REQUESTS]"
150+
echo ">> AB total runs: $TOTAL_RUN"
151+
echo ">> Output file: $OUT_FILE"
152+
echo
153+
154+
INDEX_RUN=1
155+
for TRANSACTIONS_PER_REQUEST in $TRANSACTIONS_PER_REQUEST_RANGE; do
156+
for AB_CONCURRENCY in $AB_CONCURRENCY_RANGE; do
157+
RUN_STRESS_TEST "$AGGREGATOR_ENDPOINT" "$TRANSACTIONS_PER_REQUEST" "$AB_TOTAL_REQUESTS" "$AB_CONCURRENCY" "$OUT_FILE" $INDEX_RUN "$TOTAL_RUN" "$TRANSACTIONS_FILE"
158+
INDEX_RUN=$(( INDEX_RUN + 1))
159+
done
160+
done
161+
162+
echo ">> Benchmark completed:"
163+
echo ""
164+
cat "$OUT_FILE"
165+
166+

0 commit comments

Comments
 (0)