Skip to content

Commit 2f5b785

Browse files
preinleinpreinlein
committed
Adds SMP tests for idle & ascii
1 parent 2619756 commit 2f5b785

File tree

23 files changed

+378
-0
lines changed

23 files changed

+378
-0
lines changed

‎experiments/regression/README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Regression Experiments
2+
3+
This folder defines all the regression detector experiments.
4+
5+
The regression detector is a tool provided by Single Machine Performance thru the `smp` CLI.
6+
7+
It allows us to performance test lading under different scenarios/loads.
8+
9+
IMPORTANT: Any local benchmarks should be supplemented with SMP regression detector experiments that exercise that same local benchmark test.

‎experiments/regression/cases/http_ascii_1000mib/README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# HTTP ASCII 1000 MiB/s
2+
3+
Resource usage of lading under a steady 1000 MiB/s HTTP load with ASCII payloads.
4+
5+
## What
6+
7+
Runs lading as a target with an HTTP generator sending ASCII payloads at 1000 MiB/s to its own HTTP blackhole. This exercises the HTTP generator, ASCII payload construction, and HTTP blackhole within a single lading instance at maximum throughput.
8+
9+
## Why
10+
11+
Establishes a baseline for lading's resource consumption under an extreme HTTP workload. Regressions here indicate overhead in the HTTP generator or payload path at high throughput.
12+
13+
## Paired Benchmark
14+
15+
This experiment is paired with the local benchmark in `lading_payload/benches/ascii.rs`.
16+
If throughput sizes change in either place, update the other to match.
17+
18+
## Enforcements
19+
20+
Memory usage is enforced by bounding `total_pss_bytes`.
21+
CPU usage is enforced by bounding `avg(total_cpu_usage_millicores)`.

‎experiments/regression/cases/http_ascii_1000mib/experiment.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
optimization_goal: memory
2+
erratic: false
3+
4+
target:
5+
name: lading-target
6+
# Setting experiment duration infinite here implies that lading will not go thru the shutdown process
7+
# This is acceptable as we are less concerned about the shutdown performance
8+
command: "/usr/bin/lading --no-target --experiment-duration-infinite --config-path /etc/lading-target/lading.yaml"
9+
cpu_allotment: 2
10+
memory_allotment: 1250 MiB
11+
12+
environment:
13+
DD_SERVICE: lading-target
14+
RUST_LOG: info
15+
RUST_BACKTRACE: 1
16+
17+
checks:
18+
- name: memory_usage
19+
description: "Memory usage quality gate. Bounds total memory usage."
20+
bounds:
21+
series: total_pss_bytes
22+
upper_bound: "1130 MiB"
23+
- name: cpu_usage
24+
description: "CPU usage quality gate. Bounds total average millicore usage."
25+
bounds:
26+
series: avg(total_cpu_usage_millicores)
27+
upper_bound: 400

‎experiments/regression/cases/http_ascii_1000mib/lading-target/lading.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
generator:
2+
- http:
3+
seed: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
4+
headers: {}
5+
target_uri: "http://127.0.0.1:8080/"
6+
bytes_per_second: "1000 MiB"
7+
parallel_connections: 1
8+
method:
9+
post:
10+
maximum_prebuild_cache_size_bytes: "5 GiB"
11+
variant: "ascii"
12+
13+
blackhole:
14+
- http:
15+
binding_addr: "0.0.0.0:8080"
16+
17+
telemetry:
18+
addr: "0.0.0.0:9000"

‎experiments/regression/cases/http_ascii_1000mib/lading/lading.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
generator: []
2+
3+
blackhole:
4+
- http:
5+
binding_addr: "127.0.0.1:9091"
6+
7+
target_metrics:
8+
- prometheus:
9+
uri: "http://127.0.0.1:9000/metrics"

‎experiments/regression/cases/http_ascii_100mib/README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# HTTP ASCII 100 MiB/s
2+
3+
Resource usage of lading under a steady 100 MiB/s HTTP load with ASCII payloads.
4+
5+
## What
6+
7+
Runs lading as a target with an HTTP generator sending ASCII payloads at 100 MiB/s to its own HTTP blackhole. This exercises the HTTP generator, ASCII payload construction, and HTTP blackhole within a single lading instance.
8+
9+
## Why
10+
11+
Establishes a baseline for lading's resource consumption under a high HTTP workload. Regressions here indicate overhead in the HTTP generator or payload path independent of payload complexity.
12+
13+
## Paired Benchmark
14+
15+
This experiment is paired with the local benchmark in `lading_payload/benches/ascii.rs`.
16+
If throughput sizes change in either place, update the other to match.
17+
18+
## Enforcements
19+
20+
Memory usage is enforced by bounding `total_pss_bytes`.
21+
CPU usage is enforced by bounding `avg(total_cpu_usage_millicores)`.

‎experiments/regression/cases/http_ascii_100mib/experiment.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
optimization_goal: memory
2+
erratic: false
3+
4+
target:
5+
name: lading-target
6+
# Setting experiment duration infinite here implies that lading will not go thru the shutdown process
7+
# This is acceptable as we are less concerned about the shutdown performance
8+
command: "/usr/bin/lading --no-target --experiment-duration-infinite --config-path /etc/lading-target/lading.yaml"
9+
cpu_allotment: 2
10+
memory_allotment: 630 MiB
11+
12+
environment:
13+
DD_SERVICE: lading-target
14+
RUST_LOG: info
15+
RUST_BACKTRACE: 1
16+
17+
checks:
18+
- name: memory_usage
19+
description: "Memory usage quality gate. Bounds total memory usage."
20+
bounds:
21+
series: total_pss_bytes
22+
upper_bound: "575 MiB"
23+
- name: cpu_usage
24+
description: "CPU usage quality gate. Bounds total average millicore usage."
25+
bounds:
26+
series: avg(total_cpu_usage_millicores)
27+
upper_bound: 155

‎experiments/regression/cases/http_ascii_100mib/lading-target/lading.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
generator:
2+
- http:
3+
seed: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
4+
headers: {}
5+
target_uri: "http://127.0.0.1:8080/"
6+
bytes_per_second: "100 MiB"
7+
parallel_connections: 1
8+
method:
9+
post:
10+
maximum_prebuild_cache_size_bytes: "500 MiB"
11+
variant: "ascii"
12+
13+
blackhole:
14+
- http:
15+
binding_addr: "0.0.0.0:8080"
16+
17+
telemetry:
18+
addr: "0.0.0.0:9000"

‎experiments/regression/cases/http_ascii_100mib/lading/lading.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
generator: []
2+
3+
blackhole:
4+
- http:
5+
binding_addr: "127.0.0.1:9091"
6+
7+
target_metrics:
8+
- prometheus:
9+
uri: "http://127.0.0.1:9000/metrics"

‎experiments/regression/cases/http_ascii_10mib/README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# HTTP ASCII 10 MiB/s
2+
3+
Resource usage of lading under a steady 10 MiB/s HTTP load with ASCII payloads.
4+
5+
## What
6+
7+
Runs lading as a target with an HTTP generator sending ASCII payloads at 10 MiB/s to its own HTTP blackhole. This exercises the HTTP generator, ASCII payload construction, and HTTP blackhole within a single lading instance.
8+
9+
## Why
10+
11+
Establishes a baseline for lading's resource consumption under a moderate HTTP workload. Regressions here indicate overhead in the HTTP generator or payload path independent of payload complexity.
12+
13+
## Paired Benchmark
14+
15+
This experiment is paired with the local benchmark in `lading_payload/benches/ascii.rs`.
16+
If throughput sizes change in either place, update the other to match.
17+
18+
## Enforcements
19+
20+
Memory usage is enforced by bounding `total_pss_bytes`.
21+
CPU usage is enforced by bounding `avg(total_cpu_usage_millicores)`.

0 commit comments

Comments
 (0)