Adds SMP tests for idle & ascii

experiments/regression/README.md

@@ -0,0 +1,9 @@
+# Regression Experiments
+
+This folder defines all the regression detector experiments. 
+
+The regression detector is a tool provided by Single Machine Performance thru the `smp` CLI.
+
+It allows us to performance test lading under different scenarios/loads.
+
+IMPORTANT: Any local benchmarks should be supplemented with SMP regression detector experiments that exercise that same local benchmark test.

experiments/regression/cases/http_ascii_1000mib/README.md

@@ -0,0 +1,21 @@
+# HTTP ASCII 1000 MiB/s
+
+Resource usage of lading under a steady 1000 MiB/s HTTP load with ASCII payloads.
+
+## What
+
+Runs lading as a target with an HTTP generator sending ASCII payloads at 1000 MiB/s to its own HTTP blackhole. This exercises the HTTP generator, ASCII payload construction, and HTTP blackhole within a single lading instance at maximum throughput.
+
+## Why
+
+Establishes a baseline for lading's resource consumption under an extreme HTTP workload. Regressions here indicate overhead in the HTTP generator or payload path at high throughput.
+
+## Paired Benchmark
+
+This experiment is paired with the local benchmark in `lading_payload/benches/ascii.rs`.
+If throughput sizes change in either place, update the other to match.
+
+## Enforcements
+
+Memory usage is enforced by bounding `total_pss_bytes`.
+CPU usage is enforced by bounding `avg(total_cpu_usage_millicores)`.

experiments/regression/cases/http_ascii_1000mib/experiment.yaml

@@ -0,0 +1,27 @@
+optimization_goal: memory
+erratic: false
+
+target:
+  name: lading-target
+  # Setting experiment duration infinite here implies that lading will not go thru the shutdown process
+  # This is acceptable as we are less concerned about the shutdown performance
+  command: "/usr/bin/lading --no-target --experiment-duration-infinite --config-path /etc/lading-target/lading.yaml"
+  cpu_allotment: 2
+  memory_allotment: 1250 MiB
+
+  environment:
+    DD_SERVICE: lading-target
+    RUST_LOG: info
+    RUST_BACKTRACE: 1
+
+checks:
+  - name: memory_usage
+    description: "Memory usage quality gate. Bounds total memory usage."
+    bounds:
+      series: total_pss_bytes
+      upper_bound: "1130 MiB"
+  - name: cpu_usage
+    description: "CPU usage quality gate. Bounds total average millicore usage."
+    bounds:
+      series: avg(total_cpu_usage_millicores)
+      upper_bound: 400

experiments/regression/cases/http_ascii_1000mib/lading-target/lading.yaml

@@ -0,0 +1,18 @@
+generator:
+  - http:
+      seed: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
+      headers: {}
+      target_uri: "http://127.0.0.1:8080/"
+      bytes_per_second: "1000 MiB"
+      parallel_connections: 1
+      method:
+        post:
+          maximum_prebuild_cache_size_bytes: "5 GiB"
+          variant: "ascii"
+
+blackhole:
+  - http:
+      binding_addr: "0.0.0.0:8080"
+
+telemetry:
+  addr: "0.0.0.0:9000"

experiments/regression/cases/http_ascii_1000mib/lading/lading.yaml

@@ -0,0 +1,9 @@
+generator: []
+
+blackhole:
+  - http:
+      binding_addr: "127.0.0.1:9091"
+
+target_metrics:
+  - prometheus:
+      uri: "http://127.0.0.1:9000/metrics"

experiments/regression/cases/http_ascii_100mib/README.md

@@ -0,0 +1,21 @@
+# HTTP ASCII 100 MiB/s
+
+Resource usage of lading under a steady 100 MiB/s HTTP load with ASCII payloads.
+
+## What
+
+Runs lading as a target with an HTTP generator sending ASCII payloads at 100 MiB/s to its own HTTP blackhole. This exercises the HTTP generator, ASCII payload construction, and HTTP blackhole within a single lading instance.
+
+## Why
+
+Establishes a baseline for lading's resource consumption under a high HTTP workload. Regressions here indicate overhead in the HTTP generator or payload path independent of payload complexity.
+
+## Paired Benchmark
+
+This experiment is paired with the local benchmark in `lading_payload/benches/ascii.rs`.
+If throughput sizes change in either place, update the other to match.
+
+## Enforcements
+
+Memory usage is enforced by bounding `total_pss_bytes`.
+CPU usage is enforced by bounding `avg(total_cpu_usage_millicores)`.

experiments/regression/cases/http_ascii_100mib/experiment.yaml

@@ -0,0 +1,27 @@
+optimization_goal: memory
+erratic: false
+
+target:
+  name: lading-target
+  # Setting experiment duration infinite here implies that lading will not go thru the shutdown process
+  # This is acceptable as we are less concerned about the shutdown performance
+  command: "/usr/bin/lading --no-target --experiment-duration-infinite --config-path /etc/lading-target/lading.yaml"
+  cpu_allotment: 2
+  memory_allotment: 630 MiB
+
+  environment:
+    DD_SERVICE: lading-target
+    RUST_LOG: info
+    RUST_BACKTRACE: 1
+
+checks:
+  - name: memory_usage
+    description: "Memory usage quality gate. Bounds total memory usage."
+    bounds:
+      series: total_pss_bytes
+      upper_bound: "575 MiB"
+  - name: cpu_usage
+    description: "CPU usage quality gate. Bounds total average millicore usage."
+    bounds:
+      series: avg(total_cpu_usage_millicores)
+      upper_bound: 155

experiments/regression/cases/http_ascii_100mib/lading-target/lading.yaml

@@ -0,0 +1,18 @@
+generator:
+  - http:
+      seed: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
+      headers: {}
+      target_uri: "http://127.0.0.1:8080/"
+      bytes_per_second: "100 MiB"
+      parallel_connections: 1
+      method:
+        post:
+          maximum_prebuild_cache_size_bytes: "500 MiB"
+          variant: "ascii"
+
+blackhole:
+  - http:
+      binding_addr: "0.0.0.0:8080"
+
+telemetry:
+  addr: "0.0.0.0:9000"

experiments/regression/cases/http_ascii_100mib/lading/lading.yaml

@@ -0,0 +1,9 @@
+generator: []
+
+blackhole:
+  - http:
+      binding_addr: "127.0.0.1:9091"
+
+target_metrics:
+  - prometheus:
+      uri: "http://127.0.0.1:9000/metrics"

experiments/regression/cases/http_ascii_10mib/README.md

@@ -0,0 +1,21 @@
+# HTTP ASCII 10 MiB/s
+
+Resource usage of lading under a steady 10 MiB/s HTTP load with ASCII payloads.
+
+## What
+
+Runs lading as a target with an HTTP generator sending ASCII payloads at 10 MiB/s to its own HTTP blackhole. This exercises the HTTP generator, ASCII payload construction, and HTTP blackhole within a single lading instance.
+
+## Why
+
+Establishes a baseline for lading's resource consumption under a moderate HTTP workload. Regressions here indicate overhead in the HTTP generator or payload path independent of payload complexity.
+
+## Paired Benchmark
+
+This experiment is paired with the local benchmark in `lading_payload/benches/ascii.rs`.
+If throughput sizes change in either place, update the other to match.
+
+## Enforcements
+
+Memory usage is enforced by bounding `total_pss_bytes`.
+CPU usage is enforced by bounding `avg(total_cpu_usage_millicores)`.

experiments/regression/cases/http_ascii_10mib/experiment.yaml

@@ -0,0 +1,27 @@
+optimization_goal: memory
+erratic: false
+
+target:
+  name: lading-target
+  # Setting experiment duration infinite here implies that lading will not go thru the shutdown process
+  # This is acceptable as we are less concerned about the shutdown performance
+  command: "/usr/bin/lading --no-target --experiment-duration-infinite --config-path /etc/lading-target/lading.yaml"
+  cpu_allotment: 2
+  memory_allotment: 150 MiB
+
+  environment:
+    DD_SERVICE: lading-target
+    RUST_LOG: info
+    RUST_BACKTRACE: 1
+
+checks:
+  - name: memory_usage
+    description: "Memory usage quality gate. Bounds total memory usage."
+    bounds:
+      series: total_pss_bytes
+      upper_bound: "132 MiB"
+  - name: cpu_usage
+    description: "CPU usage quality gate. Bounds total average millicore usage."
+    bounds:
+      series: avg(total_cpu_usage_millicores)
+      upper_bound: 14

experiments/regression/cases/http_ascii_10mib/lading-target/lading.yaml

@@ -0,0 +1,18 @@
+generator:
+  - http:
+      seed: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
+      headers: {}
+      target_uri: "http://127.0.0.1:8080/"
+      bytes_per_second: "10 MiB"
+      parallel_connections: 1
+      method:
+        post:
+          maximum_prebuild_cache_size_bytes: "100 MiB"
+          variant: "ascii"
+
+blackhole:
+  - http:
+      binding_addr: "0.0.0.0:8080"
+
+telemetry:
+  addr: "0.0.0.0:9000"

experiments/regression/cases/http_ascii_10mib/lading/lading.yaml

@@ -0,0 +1,9 @@
+generator: []
+
+blackhole:
+  - http:
+      binding_addr: "127.0.0.1:9091"
+
+target_metrics:
+  - prometheus:
+      uri: "http://127.0.0.1:9000/metrics"

experiments/regression/cases/http_ascii_1mib/README.md

@@ -0,0 +1,21 @@
+# HTTP ASCII 1 MiB/s
+
+Resource usage of lading under a steady 1 MiB/s HTTP load with ASCII payloads.
+
+## What
+
+Runs lading as a target with an HTTP generator sending ASCII payloads at 1 MiB/s to its own HTTP blackhole. This exercises the HTTP generator, ASCII payload construction, and HTTP blackhole within a single lading instance.
+
+## Why
+
+Establishes a baseline for lading's resource consumption under a simple, sustained HTTP workload. Regressions here indicate overhead in the HTTP generator or payload path independent of payload complexity.
+
+## Paired Benchmark
+
+This experiment is paired with the local benchmark in `lading_payload/benches/ascii.rs`.
+If throughput sizes change in either place, update the other to match.
+
+## Enforcements
+
+Memory usage is enforced by bounding `total_pss_bytes`. 
+CPU usage is enforced by bounding `avg(total_cpu_usage_millicores)`.

experiments/regression/cases/http_ascii_1mib/experiment.yaml

@@ -0,0 +1,27 @@
+optimization_goal: memory
+erratic: false
+
+target:
+  name: lading-target
+  # Setting experiment duration infinite here implies that lading will not go thru the shutdown process
+  # This is acceptable as we are less concerned about the shutdown performance
+  command: "/usr/bin/lading --no-target --experiment-duration-infinite --config-path /etc/lading-target/lading.yaml"
+  cpu_allotment: 2
+  memory_allotment: 40 MiB
+
+  environment:
+    DD_SERVICE: lading-target
+    RUST_LOG: info
+    RUST_BACKTRACE: 1
+
+checks:
+  - name: memory_usage
+    description: "Memory usage quality gate. Bounds total memory usage."
+    bounds:
+      series: total_pss_bytes
+      upper_bound: "34 MiB"
+  - name: cpu_usage
+    description: "CPU usage quality gate. Bounds total average millicore usage."
+    bounds:
+      series: avg(total_cpu_usage_millicores)
+      upper_bound: 2.4

experiments/regression/cases/http_ascii_1mib/lading-target/lading.yaml

@@ -0,0 +1,18 @@
+generator:
+  - http:
+      seed: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]
+      headers: {}
+      target_uri: "http://127.0.0.1:8080/"
+      bytes_per_second: "1 MiB"
+      parallel_connections: 1
+      method:
+        post:
+          maximum_prebuild_cache_size_bytes: "10 MiB"
+          variant: "ascii"
+
+blackhole:
+  - http:
+      binding_addr: "0.0.0.0:8080"
+
+telemetry:
+  addr: "0.0.0.0:9000"

experiments/regression/cases/http_ascii_1mib/lading/lading.yaml

@@ -0,0 +1,9 @@
+generator: []
+
+blackhole:
+  - http:
+      binding_addr: "127.0.0.1:9091"
+
+target_metrics:
+  - prometheus:
+      uri: "http://127.0.0.1:9000/metrics"

experiments/regression/cases/idle/README.md

@@ -0,0 +1,16 @@
+# Idle
+
+Baseline resource usage of lading with no active workload.
+
+## What
+
+Runs lading as a target with an empty generator and a single HTTP blackhole; this represents the most basic way to run lading.
+
+## Why
+
+Establishes a floor for lading's memory footprint. Regressions here indicate overhead introduced independent of any workload.
+
+## Enforcements
+
+Memory usage is enforced by bounding `total_pss_bytes`.
+CPU usage is enforced by bounding `avg(total_cpu_usage_millicores)`.

experiments/regression/cases/idle/experiment.yaml

@@ -0,0 +1,27 @@
+optimization_goal: memory
+erratic: false
+
+target:
+  name: lading-target
+  # Setting experiment duration infinite here implies that lading will not go thru the shutdown process
+  # This is acceptable as we are less concerned about the shutdown performance
+  command: "/usr/bin/lading --no-target --experiment-duration-infinite --config-path /etc/lading-target/lading.yaml"
+  cpu_allotment: 2
+  memory_allotment: 17 MiB
+
+  environment:
+    DD_SERVICE: lading-target
+    RUST_LOG: info
+    RUST_BACKTRACE: 1
+
+checks:
+  - name: memory_usage
+    description: "Memory usage quality gate. Bounds total memory usage."
+    bounds:
+      series: total_pss_bytes
+      upper_bound: "14 MiB"
+  - name: cpu_usage
+    description: "CPU usage quality gate. Bounds total average millicore usage."
+    bounds:
+      series: avg(total_cpu_usage_millicores)
+      upper_bound: 0.6

experiments/regression/cases/idle/lading-target/lading.yaml

@@ -0,0 +1,8 @@
+generator: []
+
+blackhole:
+  - http:
+      binding_addr: "0.0.0.0:8080"
+
+telemetry:
+  addr: "0.0.0.0:9000"

experiments/regression/cases/idle/lading/lading.yaml

@@ -0,0 +1,9 @@
+generator: []
+
+blackhole:
+  - http:
+      binding_addr: "127.0.0.1:9091"
+
+target_metrics:
+  - prometheus:
+      uri: "http://127.0.0.1:9000/metrics"

experiments/regression/config.yaml

@@ -0,0 +1,6 @@
+lading:
+  version: 0.31.2
+
+target:
+  ddprof_replicas: 0
+  internal_profiling_replicas: 0