Skip to content

Commit dbfed5a

Browse files
lrubaszepepoviolagithub-actions[bot]lazam
authored
Make zombienet CI great again (#8748)
This PR re-enables Zombienet CI tests. Changes: - Enable Zombienet Polkadot and Substrate tests - Zombienet Cumulus will be enabled separately, when being migrated to `zombienet-sdk` - Switch zombienet from `k8s` to `native` provider `k8s` turned out to be unstable for both `zombienet` and `zombienet-sdk`. Issues observed: - problem with launching a pod - pods (and thus nodes) were not spawned at the same time (differences up to 120s), which affected some tests - `kubectl` command failed Observed many times for: - kubectl cp - kubectl exec - kubectl logs If we ever want to switch back to `k8s` we must ensure above issues no longer exist. - Tweaks some tests to make sure they constantly pass Authors of those tests are kindly asked to review the changes. - Some improvements and fixes `zombienet` and `zombienet-sdk` frameworks - Assign more beefy runners for more demanding tests Rule of thumb: use large runner if spawned network consist of more than 4 nodes - Disable some tests to let their authors to stabilize them Created a dedicated `.github/zombienet-flaky-tests` file for more clarity ATM there are 16 flaky tests. Their authors are kindly asked to fix them and re-enable. --------- Co-authored-by: Javier Viola <[email protected]> Co-authored-by: Javier Viola <[email protected]> Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Anthony Lazam <[email protected]>
1 parent dc3d0e5 commit dbfed5a

File tree

24 files changed

+986
-253
lines changed

24 files changed

+986
-253
lines changed

‎.github/actions/download-artifact-extract/action.yml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
name: "Download and extract artifact"
2+
description: "Downloads an artifact, extracts it, and optionally copies files to a destination"
3+
4+
inputs:
5+
artifact-name:
6+
description: "Name of the artifact to download"
7+
required: true
8+
gh-token:
9+
description: "GITHUB_TOKEN to use for downloading artifacts"
10+
required: true
11+
run-id:
12+
description: "Run ID from which to download the artifact"
13+
required: true
14+
extract-path:
15+
description: "Path where to extract the artifact"
16+
default: "."
17+
required: false
18+
files-to-copy:
19+
description: "Comma-separated (or newline-separated, remember about |) list of files to copy from the extracted artifact"
20+
required: false
21+
destination-path:
22+
description: "Destination path for copied files"
23+
required: false
24+
cleanup:
25+
description: "Whether to remove downloaded artifacts after copying (true/false)"
26+
required: false
27+
default: "false"
28+
29+
runs:
30+
using: "composite"
31+
steps:
32+
- uses: actions/[email protected]
33+
with:
34+
name: ${{ inputs.artifact-name }}
35+
github-token: ${{ inputs.gh-token }}
36+
run-id: ${{ inputs.run-id }}
37+
path: ${{ inputs.extract-path }}
38+
39+
- name: Extract artifact
40+
shell: bash
41+
working-directory: ${{ inputs.extract-path }}
42+
run: |
43+
if [[ -f artifacts.tar ]]; then
44+
tar -xvf artifacts.tar
45+
elif [[ -f *.tar ]]; then
46+
tar -xvf *.tar
47+
elif [[ -f *.tar.gz ]]; then
48+
tar -xzvf *.tar.gz
49+
elif [[ -f *.tgz ]]; then
50+
tar -xzvf *.tgz
51+
elif [[ -f *.zip ]]; then
52+
unzip *.zip
53+
else
54+
echo "No archive file found to extract"
55+
ls -la
56+
fi
57+
58+
- name: Copy files if specified
59+
if: inputs.files-to-copy != ''
60+
env:
61+
FILES_TO_COPY: ${{ inputs.files-to-copy }}
62+
DESTINATION_PATH: ${{ inputs.destination-path }}
63+
EXTRACT_PATH: ${{ inputs.extract-path }}
64+
CLEANUP: ${{ inputs.cleanup }}
65+
66+
shell: bash
67+
run: |
68+
# Create destination directory
69+
mkdir -p "$DESTINATION_PATH"
70+
71+
echo "$FILES_TO_COPY" | tr ',' '\n' | while read -r file; do
72+
# trim leading and trailing whitespaces
73+
file="$(echo "$file" | xargs)"
74+
if [[ -n "$file" ]]; then
75+
echo "Copying $file to $DESTINATION_PATH"
76+
cp -r "$EXTRACT_PATH/$file" "$DESTINATION_PATH/"
77+
fi
78+
done
79+
80+
# Cleanup if requested
81+
if [[ "$CLEANUP" == "true" ]]; then
82+
echo "Cleaning up downloaded artifacts in $EXTRACT_PATH"
83+
rm -rf "$EXTRACT_PATH"
84+
fi

‎.github/actions/download-binaries-for-zombienet-tests/action.yml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
name: "Download binaries for zombienet tests"
2+
description: "Zombienet native tests expects some set of binaries to be available in the filesystem"
3+
4+
inputs:
5+
build-id:
6+
description: ""
7+
required: true
8+
ref-slug:
9+
description: "Ref slug (e.g branch-name-short)"
10+
required: true
11+
gh-token:
12+
description: "GITHUB_TOKEN to use for downloading artifacts"
13+
required: true
14+
destination-path:
15+
description: "Destination path for copied files"
16+
required: false
17+
18+
runs:
19+
using: "composite"
20+
steps:
21+
- uses: ./.github/actions/download-artifact-extract
22+
with:
23+
artifact-name: build-linux-substrate-${{ inputs.ref-slug }}
24+
gh-token: ${{ inputs.gh-token }}
25+
run-id: ${{ inputs.build-id }}
26+
extract-path: ./tmp
27+
files-to-copy: |
28+
artifacts/substrate/substrate
29+
destination-path: ${{ inputs.destination-path }}
30+
cleanup: "true"
31+
32+
- uses: ./.github/actions/download-artifact-extract
33+
with:
34+
artifact-name: build-linux-stable-${{ inputs.ref-slug }}
35+
gh-token: ${{ inputs.gh-token }}
36+
run-id: ${{ inputs.build-id }}
37+
extract-path: ./tmp
38+
files-to-copy: |
39+
artifacts/polkadot
40+
artifacts/polkadot-execute-worker
41+
artifacts/polkadot-prepare-worker
42+
destination-path: ${{ inputs.destination-path }}
43+
cleanup: "true"
44+
45+
46+
- uses: ./.github/actions/download-artifact-extract
47+
with:
48+
artifact-name: build-linux-stable-cumulus-${{ inputs.ref-slug }}
49+
gh-token: ${{ inputs.gh-token }}
50+
run-id: ${{ inputs.build-id }}
51+
extract-path: ./tmp
52+
files-to-copy: |
53+
artifacts/polkadot-parachain
54+
destination-path: ${{ inputs.destination-path }}
55+
cleanup: "true"
56+
57+
- uses: ./.github/actions/download-artifact-extract
58+
with:
59+
artifact-name: build-test-parachain-${{ inputs.ref-slug }}
60+
gh-token: ${{ inputs.gh-token }}
61+
run-id: ${{ inputs.build-id }}
62+
extract-path: ./tmp
63+
files-to-copy: |
64+
artifacts/test-parachain
65+
destination-path: ${{ inputs.destination-path }}
66+
cleanup: "true"
67+
68+
- uses: ./.github/actions/download-artifact-extract
69+
with:
70+
artifact-name: build-test-collators-${{ inputs.ref-slug }}
71+
gh-token: ${{ inputs.gh-token }}
72+
run-id: ${{ inputs.build-id }}
73+
extract-path: ./tmp
74+
files-to-copy: |
75+
artifacts/adder-collator
76+
artifacts/undying-collator
77+
destination-path: ${{ inputs.destination-path }}
78+
cleanup: "true"
79+
80+
- uses: ./.github/actions/download-artifact-extract
81+
with:
82+
artifact-name: build-malus-${{ inputs.ref-slug }}
83+
gh-token: ${{ inputs.gh-token }}
84+
run-id: ${{ inputs.build-id }}
85+
extract-path: ./tmp
86+
# TODO: should copy polkadot-execute-worker and polkadot-prepare-worker?
87+
# if yes then it overlaps with build-linux-stable - address this
88+
files-to-copy: |
89+
artifacts/malus
90+
destination-path: ${{ inputs.destination-path }}
91+
cleanup: "true"
92+
93+
- uses: ./.github/actions/download-artifact-extract
94+
with:
95+
artifact-name: build-templates-node-${{ inputs.ref-slug }}
96+
gh-token: ${{ inputs.gh-token }}
97+
run-id: ${{ inputs.build-id }}
98+
extract-path: ./tmp
99+
files-to-copy: |
100+
artifacts/minimal-template-node
101+
artifacts/parachain-template-node
102+
artifacts/solochain-template-node
103+
destination-path: ${{ inputs.destination-path }}
104+
cleanup: "true"

‎.github/actions/zombienet-sdk/action.yml‎

Lines changed: 58 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
name: "Zombienet-sdk test"
2+
description: "Runs zombienet-sdk tests with archived artifacts"
23
inputs:
34
build-id:
45
description: ""
56
required: true
67
ref-slug:
7-
description:
8+
description: "Ref slug (e.g branch-name-short)"
89
required: true
910
test:
1011
description: "test filter to pass to nextest (e.g: functional::spam_statement_distribution_requests::spam_statement_distribution_requests_test)"
@@ -29,35 +30,79 @@ runs:
2930
run: |
3031
echo "Vars"
3132
echo "ZOMBIENET_INTEGRATION_TEST_IMAGE: $ZOMBIENET_INTEGRATION_TEST_IMAGE"
32-
echo "COL_IMAGE: $COL_IMAGE"
33+
echo "ZOMBIE_PROVIDER": $ZOMBIE_PROVIDER
3334
echo "POLKADOT_IMAGE: $POLKADOT_IMAGE"
35+
echo "CUMULUS_IMAGE: $CUMULUS_IMAGE"
36+
echo "COL_IMAGE: $COL_IMAGE"
3437
echo "MALUS_IMAGE: $MALUS_IMAGE"
35-
echo "RUN_IN_CI: $RUN_IN_CI"
3638
echo "Inputs"
3739
echo "test: $TEST_NAME"
3840
echo "prefix: $PREFIX"
3941
40-
- uses: actions/[email protected]
42+
- name: Download binaries for zombienet native tests
43+
if: env.ZOMBIE_PROVIDER == 'native'
44+
uses: ./.github/actions/download-binaries-for-zombienet-tests
45+
with:
46+
gh-token: ${{ inputs.gh-token }}
47+
ref-slug: ${{ inputs.ref-slug }}
48+
build-id: ${{ inputs.build-id }}
49+
destination-path: ./bin
50+
51+
- uses: ./.github/actions/download-artifact-extract
4152
with:
42-
name: prepare-${{ inputs.prefix }}-zombienet-artifacts-${{ inputs.ref-slug }}
43-
github-token: ${{ inputs.gh-token }}
53+
artifact-name: prepare-${{ inputs.prefix }}-zombienet-artifacts-${{ inputs.ref-slug }}
54+
gh-token: ${{ inputs.gh-token }}
4455
run-id: ${{ inputs.build-id }}
4556

46-
- name: tar
57+
- name: k8s_auth
58+
if: env.ZOMBIE_PROVIDER == 'k8s'
4759
shell: bash
48-
run: tar -xvf artifacts.tar
60+
run: |
61+
. /home/nonroot/zombie-net/scripts/ci/run-test-local-env-manager.sh
62+
k8s_auth
4963
5064
- name: zombie_test
5165
shell: bash
5266
env:
53-
# zombienet-sdk expects RUN_IN_CI to be set when running in CI
54-
RUN_IN_CI: 1
67+
# don't retry sdk tests
68+
NEXTEST_RETRIES: 0
5569
TEST_NAME: ${{ inputs.test }}
5670
PREFIX: ${{ inputs.prefix }}
5771
run: |
58-
export ZOMBIE_K8S_CI_NAMESPACE=$(cat /data/namespace)
59-
ls -ltr ./artifacts
60-
cargo nextest run --archive-file ./artifacts/${PREFIX}-zombienet-tests.tar.zst --no-capture -- ${TEST_NAME}
72+
# RUN_IN_CI=1 shall be set only for k8s provider
73+
if [[ "$ZOMBIE_PROVIDER" == "native" ]]; then
74+
export RUN_IN_CI=0
75+
# set path to downloaded binaries
76+
export PATH=$(pwd)/bin:$PATH
77+
chmod +x $(pwd)/bin/*
78+
else
79+
export RUN_IN_CI=1
80+
# no need to check other runner variables. for k8s they shall store the same value
81+
if [[ $ZOMBIENET_SDK_DEFAULT_RUNNER == "parity-zombienet" ]]; then
82+
export ZOMBIE_K8S_CI_NAMESPACE=$(cat /data/namespace)
83+
fi
84+
fi
85+
86+
ls -ltr ./artifacts
87+
# We want to run tests sequentially, '--no-capture' ensures that.
88+
# If we want to get rid of '--no-capture' some day, please use '--test-threads 1' or NEXTEST_TEST_THREADS=1
89+
# Both options cannot coexist for cargo-nextest below v0.9.94
90+
cargo nextest run --archive-file ./artifacts/${PREFIX}-zombienet-tests.tar.zst --no-capture -- ${TEST_NAME}
91+
92+
- name: process_logs
93+
if: ${{ ! cancelled() }}
94+
shell: bash
95+
run: |
96+
echo "Processing log files"
97+
echo "::group::Logs"
98+
# do not fail the whole run if this step fails
99+
if ! ./.github/scripts/process-logs-zombienet-sdk.sh ; then
100+
echo "::endgroup::"
101+
echo "::warning ::WARNING: Failed to process logs"
102+
else
103+
echo "::endgroup::"
104+
fi
105+
61106
62107
- name: upload_logs
63108
uses: actions/upload-artifact@v4

‎.github/actions/zombienet/action.yml‎

Lines changed: 47 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
name: "Zombienet test v1"
2+
description: "Runs zombienet tests"
23
inputs:
34
test:
45
description: "test definition (zndsl file)"
@@ -10,6 +11,15 @@ inputs:
1011
description: "Concurrency to spawn nodes"
1112
default: 4
1213
required: false
14+
build-id:
15+
description: ""
16+
required: true
17+
ref-slug:
18+
description: "Ref slug (e.g branch-name-short)"
19+
required: true
20+
gh-token:
21+
description: "GITHUB_TOKEN to use for downloading artifacts"
22+
required: true
1323

1424
runs:
1525
using: "composite"
@@ -23,24 +33,56 @@ runs:
2333
run: |
2434
echo "Vars"
2535
echo "ZOMBIENET_INTEGRATION_TEST_IMAGE: $ZOMBIENET_INTEGRATION_TEST_IMAGE"
36+
echo "ZOMBIENET_PROVIDER: $ZOMBIENET_PROVIDER"
2637
echo "COL_IMAGE: $COL_IMAGE"
2738
echo "Inputs"
2839
echo "test: $TEST_NAME"
2940
echo "local-dir: $LOCAL_PATH"
3041
echo "concurrency: $CONCURRENCY"
3142
43+
- name: Download binaries for zombienet native tests
44+
if: env.ZOMBIENET_PROVIDER == 'native'
45+
uses: ./.github/actions/download-binaries-for-zombienet-tests
46+
with:
47+
gh-token: ${{ inputs.gh-token }}
48+
ref-slug: ${{ inputs.ref-slug }}
49+
build-id: ${{ inputs.build-id }}
50+
destination-path: ./bin
51+
52+
- name: k8s_auth
53+
if: env.ZOMBIENET_PROVIDER == 'k8s'
54+
shell: bash
55+
run: |
56+
. /home/nonroot/zombie-net/scripts/ci/run-test-local-env-manager.sh
57+
k8s_auth
58+
3259
- name: zombie_test
3360
shell: bash
3461
env:
3562
TEST_NAME: ${{ inputs.test }}
3663
LOCAL_PATH: ${{ inputs.local-dir }}
3764
CONCURRENCY: ${{ inputs.concurrency }}
3865
run: |
39-
export ZOMBIE_K8S_CI_NAMESPACE=$(cat /data/namespace)
40-
/home/nonroot/zombie-net/scripts/ci/run-test-local-env-manager.sh \
41-
--local-dir="$(pwd)/$LOCAL_PATH" \
42-
--concurrency=$CONCURRENCY \
43-
--test="$TEST_NAME"
66+
if [[ "$ZOMBIENET_PROVIDER" == "native" ]]; then
67+
# set path to downloaded binaries
68+
export PATH=$(pwd)/bin:$PATH
69+
chmod +x $(pwd)/bin/*
70+
71+
./.github/scripts/run-zombienet-test.sh \
72+
"$(pwd)/$LOCAL_PATH" \
73+
$CONCURRENCY \
74+
"$TEST_NAME"
75+
else
76+
# no need to check other runner variables. for k8s they shall store the same value
77+
if [[ $ZOMBIENET_DEFAULT_RUNNER == "parity-zombienet" ]]; then
78+
export ZOMBIE_K8S_CI_NAMESPACE=$(cat /data/namespace)
79+
fi
80+
81+
/home/nonroot/zombie-net/scripts/ci/run-test-local-env-manager.sh \
82+
--local-dir="$(pwd)/$LOCAL_PATH" \
83+
--concurrency=$CONCURRENCY \
84+
--test="$TEST_NAME"
85+
fi
4486
4587
- name: upload_logs
4688
uses: actions/upload-artifact@v4

0 commit comments

Comments
 (0)