Skip to content

Commit 9d696ff

Browse files
skudasovskudasov
authored
Improve docs for compat (#2482)
* fix * improve docs * remove trigger
1 parent 17e35fa commit 9d696ff

File tree

1 file changed

+203
-63
lines changed

1 file changed

+203
-63
lines changed

‎book/src/framework/compat.md‎

Lines changed: 203 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -1,104 +1,244 @@
11
# Compatibility Testing
22

3-
## Integrating Compatibility CI
3+
Compatibility testing verifies that your product remains functional when Chainlink nodes are upgraded from older versions to new ones. The `ctf compat backward` command automates this process: it rolls back to an older version, boots the environment, runs your tests, then upgrades nodes one step at a time — re-running tests after each upgrade — until the latest version is validated.
44

5-
To create CL node compatibility CI in your reporisoty follow these steps:
5+
## How the Upgrade Process Works
66

7-
1. Setup role to pull CL and JD [images](https://github.com/smartcontractkit/infra/tree/master/accounts/production/global/aws/iam/roles/gha-smartcontractkit).
8-
Add it to your secrets:
9-
```bash
10-
gh secret set MY_SECRET
11-
# enter your secret, it should be something like:
12-
# arn:aws:iam::<prod_registry>:role/gha-smartcontractkit-public-ecr <- name for your role
137
```
8+
1. Select a version sequence (e.g. v2.32.0 → v2.33.0 → v2.34.0)
9+
2. Check out the oldest Git tag (so environment + test code match that release)
10+
3. Boot environment with the oldest image → run tests (baseline)
11+
4. For each next version:
12+
a. Pull the new Docker image
13+
b. Upgrade N nodes (stop container, swap image, restart — DB volumes preserved)
14+
c. Run tests again (mixed-version cluster)
15+
5. Repeat until the latest version is fully deployed and tested
16+
```
17+
18+
Each step preserves the database volumes so node state is carried forward, exactly as it would be in a real rolling upgrade.
19+
20+
---
21+
22+
## Golden Path: Enabling Compatibility Tests in Your Repository
1423

15-
2. Copy this [pipeline](https://github.com/smartcontractkit/chainlink/blob/sot-upgrade-workflow/.github/workflows/devenv-compat.yml) to your repository
24+
### Step 1 — Create an IAM Role for ECR Access
1625

17-
3. Add nightly pipeline for your product, see `df1-compat` [example](https://github.com/smartcontractkit/chainlink/blob/sot-upgrade-workflow/.github/workflows/devenv-nightly-compat.yml#L42)
26+
Your GitHub Actions runner needs permission to pull Chainlink node and Job Distributor (JD) images from AWS ECR.
1827

19-
## Usage
28+
Request the role from your infra team. The role should follow the pattern used in [gha-smartcontractkit](https://github.com/smartcontractkit/infra/tree/master/accounts/production/global/aws/iam/roles/gha-smartcontractkit) and grant:
29+
- `ecr-public:GetAuthorizationToken` on `us-east-1` (public ECR, for CL images)
30+
- `ecr:GetAuthorizationToken` + `ecr:BatchGetImage` + `ecr:GetDownloadUrlForLayer` on `us-west-2` (private ECR, for JD images)
2031

21-
### Prerequisites
32+
### Step 2 — Add GitHub Actions Secrets
2233

23-
Authorize in our SDLC ECR registry first. Get the creds and run
34+
Add the following secrets to your repository (`Settings → Secrets and variables → Actions`):
35+
36+
| Secret name | Description | Example value |
37+
|---|---|---|
38+
| `PRODUCT_IAM_ROLE` | ARN of the IAM role that grants ECR pull access. Name it with your product name, for example CCV_IAM_ROLE | `arn:aws:iam::<account_id>:role/gha-smartcontractkit-<repo>` |
39+
| `JD_REGISTRY` | Private ECR registry ID for JD images | `<production_ecr_registry_number>.dkr.ecr.us-west-2.amazonaws.com` |
40+
| `JD_IMAGE` | Full JD image reference (used by your environment config) | `<production_ecr_registry_number>.dkr.ecr.us-west-2.amazonaws.com/job-distributor:0.12.7` |
41+
42+
Using the GitHub CLI:
2443

2544
```bash
26-
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <sdlc_ecr_registry>
45+
gh secret set CCV_IAM_ROLE # paste the IAM role ARN
46+
gh secret set JD_REGISTRY # paste the JD registry URL
47+
gh secret set JD_IMAGE # paste the JD image reference
48+
```
49+
50+
### Step 3 — Copy the Compat Pipeline
51+
52+
Copy `devenv-compat.yml` from [chainlink/sot-upgrade-workflow](https://github.com/smartcontractkit/chainlink/blob/sot-upgrade-workflow/.github/workflows/devenv-compat.yml) into your repository at `.github/workflows/devenv-compat.yml`.
53+
54+
The workflow performs the following on each run:
55+
56+
```yaml
57+
- name: Checkout code
58+
uses: actions/checkout@v5
59+
with:
60+
fetch-depth: 0 # required: ctf reads all git tags
61+
62+
- name: Authenticate to AWS ECR
63+
uses: ./.github/actions/aws-ecr-auth
64+
with:
65+
role-to-assume: ${{ secrets.CCV_IAM_ROLE }}
66+
aws-region: us-east-1
67+
registry-type: public
68+
69+
- name: Authenticate to AWS ECR (JD)
70+
uses: ./.github/actions/aws-ecr-auth
71+
with:
72+
role-to-assume: ${{ secrets.CCV_IAM_ROLE }}
73+
aws-region: us-west-2
74+
registry-type: private
75+
registries: ${{ secrets.JD_REGISTRY }}
76+
77+
- name: Run compatibility test
78+
run: |
79+
ctf compat backward \
80+
--registry <your_ecr_registry>/chainlink \
81+
--buildcmd "just cli" \
82+
--envcmd "mycli r env.toml,products/myproduct/basic.toml" \
83+
--testcmd "mycli test myproduct TestSmoke/rounds" \
84+
--strip-image-suffix v \
85+
--upgrade-nodes 2 \
86+
--versions-back 3
2787
```
2888
29-
If you don't have `ctf` CLI, download it [here](https://smartcontractkit.github.io/chainlink-testing-framework/framework/getting_started.html)
89+
`fetch-depth: 0` is required because `ctf` reads all Git tags to build the version sequence.
90+
91+
### Step 4 — Add a Nightly Trigger
92+
93+
Compatibility tests are typically run on a nightly schedule rather than on every PR. Add a nightly workflow (or a separate trigger in the same file) that points to your product configuration:
94+
95+
```yaml
96+
# .github/workflows/devenv-nightly-compat.yml
97+
on:
98+
schedule:
99+
- cron: '0 6 * * *' # 06:00 UTC every night, after the nightly CL image is built
100+
workflow_dispatch: # allow manual runs
101+
102+
jobs:
103+
compat:
104+
uses: ./.github/workflows/devenv-compat.yml
105+
secrets: inherit
106+
with:
107+
```
108+
109+
See the [chainlink nightly example](https://github.com/smartcontractkit/chainlink/blob/sot-upgrade-workflow/.github/workflows/devenv-nightly-compat.yml#L42) for a complete reference.
110+
111+
### Step 5 — Write Your Compatibility Tests
112+
113+
Your tests are plain Go tests invoked via the `--testcmd` argument. They must:
114+
115+
1. Connect to the already-running environment (do not spin up new nodes inside the test)
116+
2. Exercise the behaviour that must remain functional across versions
117+
3. Be runnable at the oldest supported version AND at every intermediate mixed-version state
30118

31-
### Testing Upgrade Sequence Locally
119+
A minimal example for [DF1](https://github.com/smartcontractkit/chainlink/blob/develop/devenv/tests/ocr2/smoke_test.go#L22):
32120

33-
We have a simple tool to check compatibility for CL node clusters. The example command will filter and sort the available tags, rollback and install the oldest version, and then begin performing automatic upgrades to verify that each subsequent version remains compatible with the previous one.
121+
The test command passed to `--testcmd` is an arbitrary shell command, so you can target any subset of tests:
34122

35-
`buildcmd`, `envcmd`, `testcmd` can be arbitrary bash commands.
123+
---
124+
125+
## Running the Upgrade Sequence Locally
126+
127+
Before pushing to CI, authenticate to the SDLC ECR registry:
128+
129+
```bash
130+
aws ecr get-login-password --region us-west-2 \
131+
| docker login --username AWS --password-stdin <sdlc_ecr_registry>
132+
```
133+
134+
Install the `ctf` CLI if you haven't already — see [Getting Started](https://smartcontractkit.github.io/chainlink-testing-framework/framework/getting_started.html).
135+
136+
### Explicit version list
137+
138+
Provide the exact versions to test in order from oldest to newest. The first ref must have a working test at that Git tag.
36139

37140
```bash
38141
ctf compat backward \
39-
--registry <sdlc_ecr_registry> \
40-
--buildcmd "just cli" \
41-
--envcmd "cl r" \
42-
--testcmd "cl test ocr2 TestSmoke/rounds" \
43-
--refs 2.32.0 \
44-
--refs 2.33.0 \
45-
--refs 2.34.0 \
46-
--refs 2.35.0 \
47-
--upgrade-nodes 3
142+
--registry <sdlc_ecr_registry> \
143+
--buildcmd "just cli" \
144+
--envcmd "mycli r env.toml" \
145+
--testcmd "mycli test myproduct TestSmoke/rounds" \
146+
--refs 2.32.0 \
147+
--refs 2.33.0 \
148+
--refs 2.34.0 \
149+
--refs 2.35.0 \
150+
--upgrade-nodes 3
48151
```
49152

50-
Keep in mind that `refs` should be present in regsitry you are testing against, the first (oldest) `ref` should also have a valid end-to-end test that works.
153+
### Automatic SemVer detection (CI mode)
51154

52-
In CI we detect SemVer tags automatically, whenever a new tag appears we select last 3, rollback to the oldest and perform upgrade process.
155+
In CI, omit `--refs` and let `ctf` detect the last N SemVer tags automatically:
53156

54157
```bash
55158
ctf compat backward \
56-
--registry <sdlc_ecr_chainlink_registry> \
57-
--buildcmd "just cli" \
58-
--envcmd "cl r env.toml,products/ocr2/basic.toml" \
59-
--testcmd "cl test ocr2 TestSmoke/rounds" \
60-
--strip-image-suffix v \
61-
--upgrade-nodes 2 \
62-
--versions-back 2
159+
--registry <sdlc_ecr_chainlink_registry> \
160+
--buildcmd "just cli" \
161+
--envcmd "mycli r env.toml,products/myproduct/basic.toml" \
162+
--testcmd "mycli test myproduct TestSmoke/rounds" \
163+
--strip-image-suffix v \
164+
--upgrade-nodes 2 \
165+
--versions-back 3
63166
```
64167

65-
In case you have multiple DONs in your product and names of nodes are different please use `--node-name-template custom-cl-node-%d` option
168+
When a new SemVer tag is pushed, the pipeline selects the last `--versions-back` tags, rolls back to the oldest, and runs the full upgrade sequence.
169+
170+
---
66171

67-
### Modelling Node Operators Cluster (WIP)
172+
## All `ctf compat backward` Flags
68173

69-
It is possible to fetch versions node operators are currently running and model DON upgrade sequence locally. When `product` is specified, `compat` will fetch the current versions from the RANE SOT data source and model the upgrade sequence for versions found on real DONs up to the latest one, each node one at a time.
174+
| Flag | Default | Description |
175+
|---|---|---|
176+
| `--registry` | `smartcontract/chainlink` | Docker image registry for Chainlink nodes |
177+
| `--refs` | _(none)_ | Explicit version list (oldest → newest). Repeat for each version. If omitted, SemVer tags are detected from git. |
178+
| `--versions-back` | `1` | How many previous SemVer tags to include when auto-detecting (used without `--refs`) |
179+
| `--upgrade-nodes` | `3` | Number of nodes to upgrade at each step |
180+
| `--don_nodes` | `5` | Total size of the DON (used with `--product` for SOT modelling) |
181+
| `--buildcmd` | `just cli` | Command to build the devenv CLI binary |
182+
| `--envcmd` | _(required)_ | Command to start the environment |
183+
| `--testcmd` | _(required)_ | Command to run the compatibility tests |
184+
| `--node-name-template` | `don-node%d` | Docker container name template for CL nodes. Use `--node-name-template custom-cl-node-%d` if your DON uses custom naming. |
185+
| `--strip-image-suffix` | _(none)_ | Strip a prefix from refs before looking them up in the registry (e.g. `v` strips the leading `v` from `v2.34.0`) |
186+
| `--include-refs` | _(none)_ | Only include refs matching these patterns (e.g. `rc,beta`) |
187+
| `--exclude-refs` | _(none)_ | Exclude refs matching these patterns (e.g. `rc,beta`) |
188+
| `--no-git-rollback` | `false` | Skip checking out the oldest Git tag. Use when your tests don't live in the same repo as the node. |
189+
| `--skip-pull` | `false` | Skip `docker pull`; use locally cached images |
190+
| `--product` | _(none)_ | Enable SOT DON modelling for this product name (see below) |
191+
| `--sot-url` | `https://rane-sot-app.main.prod.cldev.sh/v1/snapshot` | RANE SOT snapshot API URL |
192+
| `--rane-add-git-tag-prefix` | _(none)_ | Prefix to add to RANE version strings to match Git tags |
193+
194+
---
195+
196+
## Advanced: Modelling Real Node Operator DON Upgrades (WIP)
197+
198+
When `--product` is specified, `ctf compat backward` fetches the versions currently running on real DONs from the RANE SOT data source and models the upgrade sequence across actual node operator versions.
70199

71200
```bash
72201
ctf compat backward \
73-
--registry <sdlc_ecr_chainlink_registry> \
74-
--buildcmd "just cli" \
75-
--envcmd "cl r env.toml,products/ocr2/basic.toml" \
76-
--testcmd "cl test ocr2 TestSmoke/rounds" \
77-
--product data-feeds \
78-
--no-git-rollback \
79-
--don_nodes 5
202+
--registry <sdlc_ecr_chainlink_registry> \
203+
--buildcmd "just cli" \
204+
--envcmd "mycli r env.toml,products/myproduct/basic.toml" \
205+
--testcmd "mycli test myproduct TestSmoke/rounds" \
206+
--product data-feeds \
207+
--don_nodes 5
80208
```
81209

82-
The tool will check out earliest Git `ref` and setup environment and tests.
210+
`--product` field is connected with `jq .nodes[].jobs[].product` field from [SOT](https://rane-sot-app.main.prod.cldev.sh/v1/snapshot) data, find your product name there.
83211

84-
If you don't have tests on this tag you can use `--no-git-rollback` to skip the rollback step.
212+
`--no-git-rollback` can be added if your product orchestration code and tests are compatible with all these versions, otherwise, Git rollback will be performed automatically.
85213

86-
Since not all the versions from SOT are currently having corresponding Git tags or images, you can provide refs directly using `--refs` flag, useful for testing.
214+
If SOT versions do not yet have corresponding Git tags or registry images, supply them explicitly with `--refs`:
87215

88216
```bash
89217
ctf compat backward \
90-
--registry <sdlc_ecr_chainlink_registry>\
91-
--buildcmd "just cli" \
92-
--envcmd "cl r env.toml,products/ocr2/basic.toml" \
93-
--testcmd "cl test ocr2 TestSmoke/rounds" \
94-
--product data-feeds \
95-
--refs "2.36.1-rc.0" \
96-
--refs "2.36.1-beta.0" \
97-
--refs "2.36.1-beta.2" \
98-
--refs "2.37.0-rc.0" \
99-
--refs "2.37.0-beta.0" \
100-
--refs "2.38.0-rc.0" \
101-
--refs "2.38.0-beta.0" \
102-
--no-git-rollback \
103-
--don_nodes 5
218+
--registry <sdlc_ecr_chainlink_registry> \
219+
--buildcmd "just cli" \
220+
--envcmd "mycli r env.toml,products/myproduct/basic.toml" \
221+
--testcmd "mycli test myproduct TestSmoke/rounds" \
222+
--product data-feeds \
223+
--refs "2.36.1-rc.0" \
224+
--refs "2.36.1-beta.0" \
225+
--refs "2.36.1-beta.2" \
226+
--refs "2.37.0-rc.0" \
227+
--refs "2.37.0-beta.0" \
228+
--refs "2.38.0-rc.0" \
229+
--refs "2.38.0-beta.0" \
230+
--no-git-rollback \
231+
--don_nodes 5
104232
```
233+
234+
---
235+
236+
## Restoring Your Working Branch After Testing
237+
238+
`ctf compat backward` checks out old Git tags during a run. To return to your working branch:
239+
240+
```bash
241+
ctf compat restore --base_branch develop
242+
# or just:
243+
git checkout develop
244+
```

0 commit comments

Comments
 (0)