Merge branch 'main' into TT-1956-Add-support-for-E2E-docker-tests-in-Flakeguard

Author: lukaszcl

.github/workflows/framework-golden-tests.yml

@@ -26,10 +26,11 @@ jobs:
             config: smoke_limited_resources.toml
             count: 1
             timeout: 10m
-          - name: TestSuiSmoke
-            config: smoke_sui.toml
-            count: 1
-            timeout: 10m
+# Sui client starts to flake and do not respond to healthcheck properly, need to be fixed
+#          - name: TestSuiSmoke
+#            config: smoke_sui.toml
+#            count: 1
+#            timeout: 10m
           - name: TestAptosSmoke
             config: smoke_aptos.toml
             count: 1
@@ -85,7 +86,7 @@ jobs:
         with:
           go-version: 1.23
       - name: Cache Go modules
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: |
             ~/.cache/go-build

.github/workflows/lint.yaml

@@ -82,14 +82,13 @@ jobs:
       - name: Check out Code
         uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871 # v4.2.1
       - name: Install Go
-        uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/setup-go@d2f9642bcc24a73400568756f24b72c188ac7a9a # v2.3.31
+        uses: smartcontractkit/.github/actions/setup-golang@3835daadbcefcae06d12dc42a405a856c980d2cc
         with:
-          test_download_vendor_packages_command: cd ${{ matrix.project.path }} && go mod download
-          go_mod_path: ${{ matrix.project.path }}go.mod
-          cache_key_id: ctf-go-${{ matrix.project.name }}
-          cache_restore_only: 'false'
+          go-version-file: ${{ matrix.project.path }}go.mod
+          use-go-cache: true
+          go-cache-dep-path: ${{ matrix.project.path }}go.sum
       - name: golangci-lint ${{ needs.tools.outputs.golangci-lint-version }}
-        uses: golangci/golangci-lint-action@9d1e0624a798bb64f6c3cea93db47765312263dc # v5.1.0
+        uses: golangci/golangci-lint-action@051d91933864810ecd5e2ea2cfd98f6a5bca5347 # v6.3.2
         with:
           version: v${{ needs.tools.outputs.golangci-lint-version }}
           args: --out-format checkstyle:golangci-lint-report.xml
@@ -113,12 +112,11 @@ jobs:
       - name: Check out Code
         uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871 # v4.2.1
       - name: Install Go
-        uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/setup-go@d2f9642bcc24a73400568756f24b72c188ac7a9a # v2.3.31
+        uses: smartcontractkit/.github/actions/setup-golang@3835daadbcefcae06d12dc42a405a856c980d2cc
         with:
-          test_download_vendor_packages_command: cd lib && go mod download
-          go_mod_path: ./lib/go.mod
-          cache_key_id: ctf-go
-          cache_restore_only: 'false'
+          go-version-file: ./lib/go.mod
+          use-go-cache: true
+          go-cache-dep-path: ./lib/go.sum
       - name: Write Go List
         working-directory: lib
         run: go list -json -deps ./... > ../go.list

.nancy-ignore

@@ -14,4 +14,5 @@ CVE-2023-42319 # CWE-noinfo: lol... go-ethereum v1.13.8 again
 CVE-2024-10086 # Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')
 CVE-2024-51744 # CWE-755: Improper Handling of Exceptional Conditions
 CVE-2024-45338 # CWE-770: Allocation of Resources Without Limits or Throttling
-CVE-2024-45337 # CWE-863: Incorrect Authorization in golang.org/x/[email protected]
+CVE-2024-45337 # CWE-863: Incorrect Authorization in golang.org/x/[email protected]
+CVE-2025-24883 # CWE-248: Uncaught Exception

CODEOWNERS

@@ -1 +1 @@
-* @sebawo @smartcontractkit/test-tooling-team
+* @smartcontractkit/test-tooling-team

book/src/SUMMARY.md

@@ -11,7 +11,7 @@
   - [NodeSet (Local Docker builds)](./framework/nodeset_docker_rebuild.md)
   - [NodeSet Compat Environment](./framework/nodeset_compatibility.md)
   - [Creating your own components](./developing/developing_components.md)
-  - [Fork Testing](./framework/fork.md)
+  - [Asserting Logs](./developing/asserting_logs.md)
   - [Quick Contracts Deployment](./framework/quick_deployment.md)
   - [Verifying Contracts](./framework/verify.md)
   - [NodeSet with External Blockchain]()
@@ -20,6 +20,7 @@
   - [Test Configuration](./framework/test_configuration_overrides.md)
   - [Exposing Components](framework/components/state.md)
   - [Debugging Tests](framework/components/debug.md)
+  - [Debugging K8s Chaos Tests](framework/chaos/debug-k8s.md)
   - [Components Cleanup](framework/components/cleanup.md)
   - [Components Caching](framework/components/caching.md)
   - [Components Resources](framework/components/resources.md)
@@ -54,7 +55,8 @@
 - [Testing Maturity Model](framework/testing.md)
   - [Smoke]()
   - [Performance]()
-  - [Chaos]()
+  - [Chaos](./framework/chaos/chaos.md)
+  - [Fork Testing](./framework/fork.md)
 - [Interactive](framework/interactive.md)
 - [Libraries](./libraries.md)
   - [Seth](./libs/seth.md)

book/src/developing/asserting_logs.md

@@ -0,0 +1,35 @@
+# Asserting Container Logs
+
+You can either assert that CL nodes have no errors like that, we check `(CRIT|PANIC|FATAL)` levels by default for all the nodes
+
+```golang
+	in, err := framework.Load[Cfg](t)
+	require.NoError(t, err)
+	t.Cleanup(func() {
+		err := framework.SaveAndCheckLogs(t)
+		require.NoError(t, err)
+	})
+```
+
+or customize file assertions
+
+```golang
+	in, err := framework.Load[Cfg](t)
+	require.NoError(t, err)
+	t.Cleanup(func() {
+		// save all the logs to default directory "logs/docker-$test_name"
+		logs, err := framework.SaveContainerLogs(fmt.Sprintf("%s-%s", framework.DefaultCTFLogsDir, t.Name()))
+		require.NoError(t, err)
+		// check that CL nodes has no errors (CRIT|PANIC|FATAL) levels
+		err = framework.CheckCLNodeContainerErrors()
+		require.NoError(t, err)
+		// do custom assertions
+		for _, l := range logs {
+			matches, err := framework.SearchLogFile(l, " name=HeadReporter version=\\d")
+			require.NoError(t, err)
+			_ = matches
+		}
+	})
+```
+
+Full [example](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/framework/examples/myproject/smoke_logs_test.go)

book/src/framework/chaos/chaos.md

@@ -0,0 +1,65 @@
+# Chaos Testing
+
+We offer Docker and Kubernetes boilerplates designed to test the resilience of `NodeSet` and `Blockchain`, which you can customize and integrate into your pipeline.
+
+
+## Goals
+
+We recommend structuring your tests as a linear suite that applies various chaos experiments and verifies the outcomes using a load testing suite. Focus on critical user metrics, such as:
+
+- The ratio of successful responses to failed responses
+- The nth percentile of response latency
+
+Next, evaluate observability:
+
+- Ensure proper alerts are triggered during failures (manual or automated)
+- Verify the service recovers within the expected timeframe (manual or automated)
+
+In summary, the **primary** focus is on meeting user expectations and maintaining SLAs, while the **secondary** focus is on observability and making operational part smoother.
+
+
+## Docker
+
+For Docker, we utilize [Pumba](https://github.com/alexei-led/pumba) to conduct chaos experiments, including:
+
+- Container reboots
+- Network simulations (such as delays, packet loss, corruption, etc., using the tc tool)
+- Stress testing for CPU and memory usage
+
+Additionally, we offer a [resources](../../framework/components/resources.md) API that allows you to test whether your software can operate effectively in low-resource environments.
+
+You can also use [fake](../../framework/components/mocking.md) package to create HTTP chaos experiments.
+
+Given the complexity of `Kubernetes`, we recommend starting with `Docker` first. Identifying faulty behavior in your services early—such as cascading latency—can prevent more severe issues when scaling up. Addressing these problems at a smaller scale can save significant time and effort later.
+
+Check `NodeSet` + `Blockchain` template [here](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/framework/examples/myproject/chaos/chaos_docker_test.go).
+
+## Kubernetes
+
+We utilize a subset of [ChaosMesh](https://chaos-mesh.org/) experiments that can be safely executed on an isolated node group. These include:
+
+- [Pod faults](https://chaos-mesh.org/docs/simulate-pod-chaos-on-kubernetes/)
+
+- [Network faults](https://chaos-mesh.org/docs/simulate-network-chaos-on-kubernetes/) – We focus on delay and partition experiments, as others may impact pods outside the dedicated node group.
+
+- [HTTP faults](https://chaos-mesh.org/docs/simulate-http-chaos-on-kubernetes/)
+
+Check `NodeSet` + `Blockchain` template [here](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/framework/examples/myproject/chaos/chaos_k8s_test.go).
+
+## Blockchain
+
+We also offer a set of blockchain-specific experiments, which typically involve API calls to blockchain simulators to execute certain actions. These include:
+
+- Adjusting gas prices
+
+- Introducing chain reorganizations (setting a new head)
+
+- Utilizing developer APIs (e.g., Anvil)
+
+Check [gas](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/framework/examples/myproject/chaos/chaos_blockchain_evm_gas_test.go) and [reorg](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/framework/examples/myproject/chaos/chaos_blockchain_evm_reorg_test.go) examples.
+
+## Debugging
+
+To debug `Docker` applications you can just use `CTFv2` deployments.
+
+To debug `K8s` please use our [simulator](../chaos/debug-k8s.md).

book/src/framework/chaos/debug-k8s.md

@@ -0,0 +1,27 @@
+# Debugging K8s Chaos Tests
+
+This deployment is used to debug various [ChaosMesh](https://chaos-mesh.org/) with [Kind](https://kind.sigs.k8s.io/)
+
+Install [DevBox](https://www.jetify.com/devbox) and run your environment
+```
+cd infra/chaosmesh-playground
+devbox run up
+```
+
+Check the services
+```
+devbox shell
+k9s
+```
+Apply experiments (inside devbox shell)
+
+If you running it from any other shell or using `Go` don't forget to apply `kubectl config set-context kind-cm-playground` before!
+```
+kubectl apply -f manifests/latency.yaml
+```
+Debug `ChaosMesh` using `k9s`, check daemon logs.
+
+Remove the environment
+```
+devbox run down
+```

book/src/framework/components/chainlink/nodeset.md

@@ -42,6 +42,8 @@ This component requires some Blockchain to be deployed, add this to config
 Then configure NodeSet
 ```toml
 [nodeset]
+  # unique NodeSet name
+  name = "don"
   # amount of Chainlink nodes to spin up
   nodes = 5
   # Override mode: can be "all" or "each"

book/src/framework/components/state.md

@@ -44,6 +44,7 @@ One node set is enough for any kind of testing, if you need more nodes consider
 You can also define a custom set of ports for any node.
 ```toml
 [nodeset]
+  name = "don"
   nodes = 5
   override_mode = "each"