Merge branch 'main' into tt-1806-required-labels

Author: Tofel

book/README.md

@@ -1,18 +1,16 @@
-# Chainlink Testing Framework Docs
-
-We use [mdBook](https://github.com/rust-lang/mdBook) for our docs. They can be found [here](https://smartcontractkit.github.io/chainlink-testing-framework/).
-
-## Development
-
-First [install Rust](https://doc.rust-lang.org/cargo/getting-started/installation.html), then [mdbook](https://github.com/rust-lang/mdBook).
-
-```sh
-# Install mdBook
+## MDBook
+Install [mdbook](https://github.com/rust-lang/mdBook)
+```
 cargo install mdbook && \
 cargo install mdbook-alerts && \
-cargo install mdbook-cmdrun
-# Run the mdBook
+cargo install mdbook-cmdrun &&
+cargo install mdbook-mermaid
+```
+
+Run it locally
+```
 make run
 ```
+Open your browser [here](http://localhost:9999/)
 
-Visit [localhost:9999](http://localhost:9999) to see your local version.
+GitHub hosted [version](https://smartcontractkit.github.io/chainlink-testing-framework/overview.html)

book/book.toml

@@ -10,6 +10,10 @@ build-dir = "docs"
 
 [output.html]
 default-theme = "ayu"
+additional-js = ["mermaid.min.js", "mermaid-init.js"]
 
 [preprocessor.alerts]
-[preprocessor.cmdrun]
+[preprocessor.cmdrun]
+
+[preprocessor.mermaid]
+command = "mdbook-mermaid"

book/mermaid-init.js

@@ -0,0 +1,35 @@
+(() => {
+    const darkThemes = ['ayu', 'navy', 'coal'];
+    const lightThemes = ['light', 'rust'];
+
+    const classList = document.getElementsByTagName('html')[0].classList;
+
+    let lastThemeWasLight = true;
+    for (const cssClass of classList) {
+        if (darkThemes.includes(cssClass)) {
+            lastThemeWasLight = false;
+            break;
+        }
+    }
+
+    const theme = lastThemeWasLight ? 'default' : 'dark';
+    mermaid.initialize({ startOnLoad: true, theme });
+
+    // Simplest way to make mermaid re-render the diagrams in the new theme is via refreshing the page
+
+    for (const darkTheme of darkThemes) {
+        document.getElementById(darkTheme).addEventListener('click', () => {
+            if (lastThemeWasLight) {
+                window.location.reload();
+            }
+        });
+    }
+
+    for (const lightTheme of lightThemes) {
+        document.getElementById(lightTheme).addEventListener('click', () => {
+            if (!lastThemeWasLight) {
+                window.location.reload();
+            }
+        });
+    }
+})();

book/mermaid.min.js

@@ -49,7 +49,33 @@
 - [Interactive](framework/interactive.md)
 - [Libraries](./libraries.md)
   - [Seth](./libs/seth.md)
-  - [WASP](./libs/wasp.md)
+  - [WASP](./libs/wasp/overview.md)
+    - [Getting started](./libs/wasp/getting_started.md)
+    - [First test (RPS test)](./libs/wasp/first_test.md)
+    - [Stateful test](./libs/wasp/stateful_test.md)
+    - [User Journey test](./libs/wasp/user_journey_test.md)
+    - [Profile test](./libs/wasp/profile_test.md)
+    - [Testing alerts]()
+    - [Configuration](./libs/wasp/configuration.md)
+    - [k8s](./libs/wasp/k8s.md)
+    - [Components](./libs/wasp/components/overview.md)
+      - [Alert Checker]()
+      - [Dashboard](./libs/wasp/components/dashboard.md)
+      - [Generator](./libs/wasp/components/generator.md)
+      - [Loki](./libs/wasp/components/loki.md)
+      - [Profile](./libs/wasp/components/profile.md)
+      - [Sampler](./libs/wasp/components/sampler.md)
+      - [Schedule](./libs/wasp/components/schedule.md)
+    - [How to](./libs/wasp/how-to/overview.md)
+      - [Start local observability stack](./libs/wasp/how-to/start_local_observability_stack.md)
+      - [Try it out quickly](./libs/wasp/how-to/run_included_tests.md)
+      - [Chose between RPS and VUs](./libs/wasp/how-to/chose_rps_vu.md)
+      - [Define NFRs and check alerts](./libs/wasp/how-to/define_nfr_check_alerts.md)
+      - [Use labels](./libs/wasp/how-to/use_labels.md)
+      - [Incorporate load tests in your workflow](./libs/wasp/how-to/incorporate_load_tests.md)
+      - [Reuse dashboard components](./libs/wasp/how-to/reuse_dashboard_components.md) 
+      - [Parallelize load](./libs/wasp/how-to/parallelise_load.md)
+      - [Debug Loki errors](./libs/wasp/how-to/debug_loki_errors.md)
   - [Havoc](./libs/havoc.md)
   - [K8s Test Runner](k8s-test-runner/k8s-test-runner.md)
 

book/src/SUMMARY.md

@@ -0,0 +1 @@
+# Try it out quickly

book/src/lib/wasp/how-to/run_included_tests.md

@@ -0,0 +1,20 @@
+# WASP - Alert Checker
+
+> [!WARNING]  
+> The API used by `AlertChecker` is unstable, and this section may be out of date.
+
+`AlertChecker` is a simple yet powerful component that allows you to verify which alerts fired during test execution.
+
+It supports the following functionalities:
+* Checking if **any alert** fired during a specified time range for a given dashboard.
+* Checking if **any group of alerts** fired for a given dashboard.
+
+The first mode is more suitable for existing dashboards, while the second mode is ideal for dashboards created specifically for the test (as it does not support time range selection).
+
+> [!NOTE]  
+> To use this component, you need to set certain Grafana-specific variables as described in the [Configuration](../configuration.md) section.
+
+For a practical example of how to use `AlertChecker`, refer to the [Testing Alerts](../testing_alerts.md) section.
+
+> [!WARNING]  
+> If you define alerts yourself, ensure they are grouped by adding a label named `requirement_name` with a value representing the alert group.

book/src/libs/wasp/components/alert_checker.md

@@ -0,0 +1,77 @@
+# WASP - Dashboard
+
+> [!WARNING]  
+> The API used to check and create alerts is unstable, and the information related to them may be out of date.
+
+
+WASP comes with a built-in dashboard that allows you to monitor test runs in real-time.  
+The dashboard includes several built-in metrics that integrate seamlessly with the `AlertChecker` component.
+
+It is built using the [Grabana](https://pkg.go.dev/github.com/K-Phoen/grabana) library, which you can use to further customize and extend the dashboard by adding your own rows and panels.
+
+> [!NOTE]  
+> To create new dashboards, you need to set certain dashboard-specific variables as described in the [Configuration](../configuration.md) section.
+
+---
+
+### Predefined Alerts
+
+WASP comes with predefined alerts for:
+* **99th percentile of the response time**
+* **Response errors**
+* **Response timeouts**
+
+You can use these predefined metrics to add simple alerts to your dashboard for conditions such as:
+* Values above or below a threshold
+* Averages above or below a threshold
+* Percentages of the total above or below a threshold
+
+For a complete list of available conditions, refer to Grabana's [ConditionEvaluator](https://pkg.go.dev/github.com/K-Phoen/[email protected]/alert#ConditionEvaluator).
+
+---
+
+### Custom Alerts
+
+Custom alerts can be composed of:
+* The simple conditions mentioned above
+* Arbitrary Loki queries
+
+Custom alerts use Grabana's [timeseries.Alert](https://pkg.go.dev/github.com/K-Phoen/[email protected]/timeseries#Alert) and must be timeseries-based.
+
+> [!NOTE]  
+> Adding a built-in alert will also add a new row to the dashboard to display the monitored metric.  
+> In contrast, custom alerts do not automatically add rows to the dashboard to prevent clutter.
+
+Each generator has its own metrics, matched by the generator name.
+
+---
+
+### Default Dashboard Panels
+
+The default dashboard includes the following panels:
+* Current RPS/VUs (depending on the generator)
+* Responses per second
+* Total successful requests
+* Total failed requests
+* Total timeout requests
+* RPS/VUs per schedule segment
+* Responses per second
+* Latency quantiles over groups (p99, p90, p50)
+* Response latencies over time
+* Logs size per second
+* Sampling statistics
+* Failed & timed-out responses
+* Logs of the statistics-pushing service
+
+![Dashboard](../images/dashboard_basic.png)
+
+Where applicable, these panels group results by generator name (`gen_name` label) and call group (`call_group` label).
+
+> [!NOTE]  
+> You can read more about using labels in the [Use labels](../how-to/use_labels.md) section.
+
+---
+
+### Creating a New Dashboard with Alerts
+
+For a practical example of how to create a new dashboard with alerts, see the [Testing Alerts](../testing_alerts.md) section.

book/src/libs/wasp/components/dashboard.md

@@ -0,0 +1,58 @@
+# WASP - Generator
+
+A **Generator** is a component that encapsulates all the characteristics of load generation, including:
+* Load type:
+  * RPS (requests per second)
+  * VUs (virtual users)
+* [Load schedule](./schedule.md)
+* Call logic
+* Response data
+* [Sampling](./sampler.md)
+* Timeouts
+
+> [!WARNING]  
+> RPS load type can only be used with a `Gun`, while VUs can only be used with a `VirtualUser`.
+
+---
+
+### Choosing Between `Gun` and `VirtualUser`
+
+#### **`Gun`**
+* Best for **stateless protocols** (e.g., HTTP).
+* Simplistic in nature; ideal for executing a single operation that does not require setup or teardown.
+* Operates using an **open model**, where:
+  * The number of requests is fixed.
+  * The load adjusts to meet the target RPS, regardless of the system's response time.
+  * There's no feedback from the system.
+* Recommended for scenarios focused on measuring **throughput**.
+
+#### **`VirtualUser`**
+* Designed for **stateful protocols** (e.g., `WebSocket`) or workflows involving multiple operations (e.g., authenticating, executing tasks, and logging out).
+* More complex, with dedicated methods for setup and teardown.
+* Operates using a **closed model**, where:
+  * New iterations start only after the previous one completes.
+  * The RPS fluctuates based on the system's response time. Longer response times reduce RPS.
+  * Feedback from the system is used to adjust the load.
+---
+
+### Closed vs. Open Models
+
+* A `Gun` follows an **open model**:
+  - It controls the rate of requests being sent.
+  - The system's response time does not impact the load generation rate.
+
+* A `VirtualUser` follows a **closed model**:
+  - It controls the rate of receiving responses.
+  - The system's response time directly impacts the load generation rate. If the system slows down, iterations take longer, reducing the RPS.
+
+---
+
+### Summary
+
+In simpler terms:
+* A **`Gun`** limits the load during the **sending** phase, making it ideal for throughput measurements.
+* A **`VirtualUser`** limits the load during the **receiving** phase, reflecting the system's performance under load.
+
+---
+
+This distinction helps you decide which tool to use based on the protocol type and the goals of your test.

book/src/libs/wasp/components/generator.md

@@ -0,0 +1,16 @@
+# WASP - Loki
+
+Loki is a component responsible for pushing batches of statistics to a Loki instance. It operates in the background using a `promtail` client.
+
+Key features include:
+* Optional basic authentication support.
+* Configurable batch sizes.
+* Support for configurable backoff retries, among others.
+
+By default, a test will fail on the first error encountered while pushing data. You can modify this behavior by setting the maximum allowed errors:
+* Setting the value to `-1` disables error checks entirely.
+
+> [!NOTE]  
+> While it is technically possible to execute a WASP test without Loki, doing so means you won't have access to load-generation-related metrics.  
+> Unless your sole interest is in the metrics sent by your application, it is highly recommended to use Loki.  
+> For this reason, Loki is considered an integral part of the WASP stack.