multi-don examples (#1614)

multi-nodeset setup + chaos testing examples

Author: skudasov

book/src/SUMMARY.md

@@ -12,7 +12,6 @@
   - [NodeSet Compat Environment](./framework/nodeset_compatibility.md)
   - [Creating your own components](./developing/developing_components.md)
   - [Asserting Logs](./developing/asserting_logs.md)
-  - [Fork Testing](./framework/fork.md)
   - [Quick Contracts Deployment](./framework/quick_deployment.md)
   - [Verifying Contracts](./framework/verify.md)
   - [NodeSet with External Blockchain]()
@@ -55,7 +54,8 @@
 - [Testing Maturity Model](framework/testing.md)
   - [Smoke]()
   - [Performance]()
-  - [Chaos]()
+  - [Chaos](./framework/chaos/chaos.md)
+  - [Fork Testing](./framework/fork.md)
 - [Interactive](framework/interactive.md)
 - [Libraries](./libraries.md)
   - [Seth](./libs/seth.md)

book/src/framework/chaos/chaos.md

@@ -0,0 +1,59 @@
+# Chaos Testing
+
+We offer Docker and Kubernetes boilerplates designed to test the resilience of `NodeSet` and `Blockchain`, which you can customize and integrate into your pipeline.
+
+
+## Goals
+
+We recommend structuring your tests as a linear suite that applies various chaos experiments and verifies the outcomes using a [load](../../libs/wasp.md) testing suite. Focus on critical user metrics, such as:
+
+- The ratio of successful responses to failed responses
+- The nth percentile of response latency
+
+Next, evaluate observability:
+
+- Ensure proper alerts are triggered during failures (manual or automated)
+- Verify the service recovers within the expected timeframe (manual or automated)
+
+In summary, the **primary** focus is on meeting user expectations and maintaining SLAs, while the **secondary** focus is on observability and making operational part smoother.
+
+
+## Docker
+
+For Docker, we utilize [Pumba](https://github.com/alexei-led/pumba) to conduct chaos experiments, including:
+
+- Container reboots
+- Network simulations (such as delays, packet loss, corruption, etc., using the tc tool)
+- Stress testing for CPU and memory usage
+
+Additionally, we offer a [resources](../../framework/components/resources.md) API that allows you to test whether your software can operate effectively in low-resource environments.
+
+You can also use [fake](../../framework/components/mocking.md) package to create HTTP chaos experiments.
+
+Given the complexity of `Kubernetes`, we recommend starting with `Docker` first. Identifying faulty behavior in your services early—such as cascading latency—can prevent more severe issues when scaling up. Addressing these problems at a smaller scale can save significant time and effort later.
+
+Check `NodeSet` + `Blockchain` template [here]().
+
+## Kubernetes
+
+We utilize a subset of [ChaosMesh](https://chaos-mesh.org/) experiments that can be safely executed on an isolated node group. These include:
+
+- [Pod faults](https://chaos-mesh.org/docs/simulate-pod-chaos-on-kubernetes/)
+
+- [Network faults](https://chaos-mesh.org/docs/simulate-network-chaos-on-kubernetes/) – We focus on delay and partition experiments, as others may impact pods outside the dedicated node group.
+
+- [HTTP faults](https://chaos-mesh.org/docs/simulate-http-chaos-on-kubernetes/)
+
+Check `NodeSet` + `Blockchain` template [here]().
+
+## Blockchain
+
+We also offer a set of blockchain-specific experiments, which typically involve API calls to blockchain simulators to execute certain actions. These include:
+
+- Adjusting gas prices
+
+- Introducing chain reorganizations (setting a new head)
+
+- Utilizing developer APIs (e.g., Anvil)
+
+Check [gas]() and [reorg]() examples.

book/src/framework/components/chainlink/nodeset.md

@@ -42,6 +42,8 @@ This component requires some Blockchain to be deployed, add this to config
 Then configure NodeSet
 ```toml
 [nodeset]
+  # unique NodeSet name
+  name = "don"
   # amount of Chainlink nodes to spin up
   nodes = 5
   # Override mode: can be "all" or "each"

book/src/framework/components/state.md

@@ -44,6 +44,7 @@ One node set is enough for any kind of testing, if you need more nodes consider
 You can also define a custom set of ports for any node.
 ```toml
 [nodeset]
+  name = "don"
   nodes = 5
   override_mode = "each"
 

book/src/framework/nodeset_capabilities.md

@@ -21,6 +21,7 @@ Create a configuration file `smoke.toml`
   docker_cmd_params = ["-b", "1"]
 
 [nodeset]
+  name = "don"
   nodes = 5
   override_mode = "all"
 

book/src/framework/nodeset_compatibility.md

@@ -9,6 +9,7 @@ Create a configuration file `smoke.toml`
   docker_cmd_params = ["-b", "1"]
 
 [nodeset]
+  name = "don"
   nodes = 5
   override_mode = "each"
 

book/src/framework/nodeset_docker_rebuild.md

@@ -9,6 +9,7 @@ Create a configuration file `smoke.toml`
   docker_cmd_params = ["-b", "1"]
 
 [nodeset]
+  name = "don"
   nodes = 5
   override_mode = "all"
 

book/src/framework/nodeset_environment.md

@@ -9,6 +9,7 @@ Create a configuration file `smoke.toml`
   docker_cmd_params = ["-b", "1"]
 
 [nodeset]
+  name = "don"
   nodes = 5
   override_mode = "all"
 

framework/.changeset/v0.5.0.md

@@ -0,0 +1,2 @@
+- Allow multiple node sets simultaneously
+- Add chaos examples for gas, reorg, k8s and docker

framework/cmd/docker.go

@@ -14,7 +14,7 @@ func removeTestContainers() error {
 	cmd := exec.Command("bash", "-c", `
 		docker ps -aq --filter "label=framework=ctf" | xargs -r docker rm -f && \
 		docker network ls --filter "label=framework=ctf" -q | xargs -r docker network rm && \
-		docker volume rm postgresql_data || true
+		docker volume ls -q | xargs -r docker volume rm || true
 	`)
 	framework.L.Debug().Msg("Running command")
 	if framework.L.GetLevel() == zerolog.DebugLevel {