Skip to content

Commit f45899b

Browse files
authored
End-to-End System Level Testing Maturity Model (#1332)
maturity model for e2e
1 parent 1778cb7 commit f45899b

File tree

10 files changed

+148
-103
lines changed

10 files changed

+148
-103
lines changed

.github/workflows/framework-golden-tests.yml

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,10 @@ jobs:
2222
config: smoke.toml
2323
count: 1
2424
timeout: 10m
25+
- name: TestUpgrade
26+
config: upgrade.toml
27+
count: 1
28+
timeout: 10m
2529
- name: TestPerformanceBaseline
2630
config: performance_baseline.toml
2731
count: 1
@@ -30,10 +34,11 @@ jobs:
3034
config: chaos.toml
3135
count: 1
3236
timeout: 10m
33-
- name: TestUpgrade
34-
config: upgrade.toml
37+
- name: TestScalability
38+
config: scalability.toml
3539
count: 1
3640
timeout: 10m
41+
name: ${{ matrix.test.name }}
3742
steps:
3843
- name: Checkout repo
3944
uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4.1.4

book/src/framework/testing.md

Lines changed: 2 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,5 @@
11
# Testing Maturity Model
22

3-
[Here](https://github.com/smartcontractkit/chainlink-testing-framework/actions/runs/11739154666/job/32703095118?pr=1311) are our "golden" templates for end-to-end tests, covering every test type:
3+
Read this [doc](https://github.com/smartcontractkit/chainlink-testing-framework/tree/main/framework/examples/myproject) to understand the rationale behind our testing approach and to explore the stages of maturity in end-to-end testing.
44

5-
- `Smoke`
6-
- `PerformanceBaseline`
7-
- `Chaos`
8-
- `Upgrade`
9-
10-
These tests act as a maturity model and are implemented across all our products.
11-
12-
Refer to this README to understand the rationale behind our testing approach and to explore the stages of maturity in end-to-end testing.
13-
14-
## Developing
15-
Run the tests locally
16-
```
17-
CTF_CONFIGS=smoke.toml go test -v -run TestSmoke
18-
CTF_CONFIGS=performance_baseline.toml go test -v -run TestPerformanceBaseline
19-
CTF_CONFIGS=chaos.toml go test -v -run TestChaos
20-
CTF_CONFIGS=upgrade.toml go test -v -run TestUpgrade
21-
```
5+
The following chapters detail specific testing types and best practices for system-level end-to-end testing.

framework/.changeset/v0.2.2.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
- Switch to port checking when we boot Chainlink nodes (eliminates chaos flakes)
2+
- Add initial e2e maturity model pipeline
Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,54 @@
11
# End-to-End Testing Project Maturity Model
22

3+
[![Smoke](https://img.shields.io/badge/Level_1-Smoke-blue?branch=maturity-model&job=TestSmoke)](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml)
4+
[![Upgrade](https://img.shields.io/badge/Level_2-Upgrade-blue?branch=maturity-model&job=TestSmoke)](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml)
5+
[![Performance Baseline](https://img.shields.io/badge/Level_3-Performance_baseline-blue?branch=maturity-model&job=TestSmoke)](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml)
6+
[![Resiliency](https://img.shields.io/badge/Level_4-Resiliency-blue?branch=maturity-model&job=TestSmoke)](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml)
7+
[![Scalability](https://img.shields.io/badge/Level_5-Scalability-blue?branch=maturity-model&job=TestSmoke)](https://github.com/smartcontractkit/chainlink-testing-framework/actions/workflows/framework-golden-tests.yml)
8+
9+
## Level 1
10+
The team maintains a system-level smoke test where all components are deployed using `docker`.
11+
12+
All on-chain changes are done through [chainlink-deployments](https://github.com/smartcontractkit/chainlink-deployments).
13+
14+
The test is readable, and the README clearly explains its purpose.
15+
16+
The test is reliable and stable when run with a `-count 30`.
17+
18+
If your project includes multiple use cases and functionality suitable for end-to-end testing, you can add additional tests at this level.
19+
20+
## Level 2
21+
The team has an "upgrade" test to verify product compatibility with older versions.
22+
23+
While the number of compatible versions is team-determined, identifying incompatibilities at the system level early is a valuable, mature practice.
24+
25+
This test deploys specific platform and plugin versions, performs an end-to-end smoke test, and then upgrades (or migrates) the plugin(s) or platform on the same database to ensure that users remain unaffected by the upgrade.
26+
27+
## Level 3
28+
The team has a baseline performance testing suite.
29+
30+
At this level, the focus is not on improving performance but on detecting any performance degradation, supported by a reliable CI pipeline.
31+
32+
This pipeline runs as needed—nightly or before releases—enabling early detection of performance issues across all critical on-chain and off-chain functionality.
33+
34+
This stage combines performance testing with observability enhancements. The team should have fundamental system-level performance tests in place, along with dashboards to monitor product-specific metrics.
35+
36+
## Level 4
37+
The team incorporates chaos engineering practices to test the system’s failure modes.
38+
39+
This stage builds on [Level 3](#level-3), as it not only verifies that the system is reliable and can recover from reasonable failures but also ensures an understanding of how these failures impact performance and user experience.
40+
41+
## Level 5
42+
The team has complete ownership of their persistent staging environment.
43+
44+
They can perform upgrades, data migrations, and run advanced load tests to validate the scalability of their applications.
45+
346
## Developing
447
Run the tests locally
548
```
649
CTF_CONFIGS=smoke.toml go test -v -run TestSmoke
50+
CTF_CONFIGS=upgrade.toml go test -v -run TestUpgrade
751
CTF_CONFIGS=performance_baseline.toml go test -v -run TestPerformanceBaseline
852
CTF_CONFIGS=chaos.toml go test -v -run TestChaos
9-
CTF_CONFIGS=upgrade.toml go test -v -run TestUpgrade
53+
CTF_CONFIGS=scalability.toml go test -v -run TestScalability
1054
```

framework/examples/myproject/gun.go renamed to framework/examples/myproject/generators/gun.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
package examples
1+
package generators
22

33
import (
44
"github.com/smartcontractkit/chainlink-testing-framework/framework/clclient"

framework/examples/myproject/go.mod

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ require (
2727
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 // indirect
2828
github.com/armon/go-metrics v0.4.1 // indirect
2929
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
30-
github.com/atotto/clipboard v0.1.4 // indirect
3130
github.com/aws/aws-sdk-go v1.45.25 // indirect
3231
github.com/aws/aws-sdk-go-v2 v1.31.0 // indirect
3332
github.com/aws/aws-sdk-go-v2/config v1.27.39 // indirect
@@ -43,26 +42,16 @@ require (
4342
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.27.3 // indirect
4443
github.com/aws/aws-sdk-go-v2/service/sts v1.31.3 // indirect
4544
github.com/aws/smithy-go v1.21.0 // indirect
46-
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
4745
github.com/benbjohnson/clock v1.3.5 // indirect
4846
github.com/beorn7/perks v1.0.1 // indirect
4947
github.com/buger/jsonparser v1.1.1 // indirect
5048
github.com/bytedance/sonic v1.12.3 // indirect
5149
github.com/bytedance/sonic/loader v0.2.0 // indirect
5250
github.com/c2h5oh/datasize v0.0.0-20220606134207-859f65c6625b // indirect
5351
github.com/c9s/goprocinfo v0.0.0-20210130143923-c95fcf8c64a8 // indirect
54-
github.com/catppuccin/go v0.2.0 // indirect
5552
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
5653
github.com/cespare/xxhash v1.1.0 // indirect
5754
github.com/cespare/xxhash/v2 v2.3.0 // indirect
58-
github.com/charmbracelet/bubbles v0.20.0 // indirect
59-
github.com/charmbracelet/bubbletea v1.1.1 // indirect
60-
github.com/charmbracelet/huh v0.6.0 // indirect
61-
github.com/charmbracelet/huh/spinner v0.0.0-20241028115900-20a4d21717a8 // indirect
62-
github.com/charmbracelet/lipgloss v0.13.0 // indirect
63-
github.com/charmbracelet/x/ansi v0.2.3 // indirect
64-
github.com/charmbracelet/x/exp/strings v0.0.0-20240722160745-212f7b056ed0 // indirect
65-
github.com/charmbracelet/x/term v0.2.0 // indirect
6655
github.com/cloudwego/base64x v0.1.4 // indirect
6756
github.com/cloudwego/iasm v0.2.0 // indirect
6857
github.com/coder/websocket v1.8.12 // indirect
@@ -72,7 +61,6 @@ require (
7261
github.com/coreos/go-semver v0.3.0 // indirect
7362
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
7463
github.com/cpuguy83/dockercfg v0.3.1 // indirect
75-
github.com/cpuguy83/go-md2man/v2 v2.0.5 // indirect
7664
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
7765
github.com/dennwc/varint v1.0.0 // indirect
7866
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
@@ -83,7 +71,6 @@ require (
8371
github.com/dustin/go-humanize v1.0.1 // indirect
8472
github.com/edsrzf/mmap-go v1.1.0 // indirect
8573
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
86-
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
8774
github.com/ethereum/go-ethereum v1.14.11 // indirect
8875
github.com/facette/natsort v0.0.0-20181210072756-2cd4dd1e2dcb // indirect
8976
github.com/fatih/color v1.16.0 // indirect
@@ -155,18 +142,14 @@ require (
155142
github.com/klauspost/cpuid/v2 v2.2.8 // indirect
156143
github.com/kylelemons/godebug v1.1.0 // indirect
157144
github.com/leodido/go-urn v1.4.0 // indirect
158-
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
159145
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
160146
github.com/magiconair/properties v1.8.7 // indirect
161147
github.com/mailru/easyjson v0.7.7 // indirect
162148
github.com/mattn/go-colorable v0.1.13 // indirect
163149
github.com/mattn/go-isatty v0.0.20 // indirect
164-
github.com/mattn/go-localereader v0.0.1 // indirect
165-
github.com/mattn/go-runewidth v0.0.16 // indirect
166150
github.com/miekg/dns v1.1.56 // indirect
167151
github.com/mitchellh/copystructure v1.0.0 // indirect
168152
github.com/mitchellh/go-homedir v1.1.0 // indirect
169-
github.com/mitchellh/hashstructure/v2 v2.0.2 // indirect
170153
github.com/mitchellh/mapstructure v1.5.0 // indirect
171154
github.com/mitchellh/reflectwalk v1.0.1 // indirect
172155
github.com/moby/docker-image-spec v1.3.1 // indirect
@@ -177,9 +160,6 @@ require (
177160
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
178161
github.com/modern-go/reflect2 v1.0.2 // indirect
179162
github.com/morikuni/aec v1.0.0 // indirect
180-
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
181-
github.com/muesli/cancelreader v0.2.2 // indirect
182-
github.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a // indirect
183163
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
184164
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect
185165
github.com/oklog/ulid v1.3.1 // indirect
@@ -189,7 +169,6 @@ require (
189169
github.com/opentracing-contrib/go-stdlib v1.0.0 // indirect
190170
github.com/opentracing/opentracing-go v1.2.0 // indirect
191171
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58 // indirect
192-
github.com/pelletier/go-toml v1.9.5 // indirect
193172
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
194173
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
195174
github.com/pkg/errors v0.9.1 // indirect
@@ -203,9 +182,7 @@ require (
203182
github.com/prometheus/exporter-toolkit v0.10.1-0.20230714054209-2f4150c63f97 // indirect
204183
github.com/prometheus/procfs v0.15.1 // indirect
205184
github.com/prometheus/prometheus v0.47.2-0.20231010075449-4b9c19fe5510 // indirect
206-
github.com/rivo/uniseg v0.4.7 // indirect
207185
github.com/rs/zerolog v1.33.0 // indirect
208-
github.com/russross/blackfriday/v2 v2.1.0 // indirect
209186
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 // indirect
210187
github.com/sercand/kuberesolver/v5 v5.1.1 // indirect
211188
github.com/shirou/gopsutil/v3 v3.23.12 // indirect
@@ -225,9 +202,7 @@ require (
225202
github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
226203
github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
227204
github.com/ugorji/go/codec v1.2.12 // indirect
228-
github.com/urfave/cli/v2 v2.27.5 // indirect
229205
github.com/x448/float16 v0.8.4 // indirect
230-
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 // indirect
231206
github.com/yusufpapurcu/wmi v1.2.3 // indirect
232207
go.etcd.io/etcd/api/v3 v3.5.7 // indirect
233208
go.etcd.io/etcd/client/pkg/v3 v3.5.7 // indirect

0 commit comments

Comments
 (0)