Skip to content

Commit e7e9a65

Browse files
spetrosispetrosi
authored
ci: Add test plan that runs CI tests and customize it for each role (#513)
* ci: Add test plan that runs CI tests and customize it for each role * Calculate number of managed nodes with this formula: (( number_of_test_playbooks / 10 + 1 )) * Add README explaining how to run the plan locally and remotely Signed-off-by: Sergei Petrosian <[email protected]> * ci: Add test plan that runs CI tests and customize it for each role * Calculate number of managed nodes with this formula: (( number_of_test_playbooks / 10 + 1 )) * Add README explaining how to run the plan locally and remotely Signed-off-by: Sergei Petrosian <[email protected]> --------- Signed-off-by: Sergei Petrosian <[email protected]>
1 parent 84cc4f4 commit e7e9a65

File tree

4 files changed

+171
-3
lines changed

4 files changed

+171
-3
lines changed

‎.codespellrc‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
[codespell]
22
check-hidden = true
3+
# Note that `-w` doesn't work when ignore-multiline-regex is set
34
# https://github.com/codespell-project/codespell/issues/3642
5+
ignore-multiline-regex = codespell:ignore-begin.*codespell:ignore-end
46
ignore-words = .codespell_ignores
57
# skip-file is not available https://github.com/codespell-project/codespell/pull/2759
8+
# .pandoc_template.html5 contains a typo in Licence that we shouldn't edit
9+
# .README.html is generated from README.md automatically - no need to check spelling
610
skip = .pandoc_template.html5,.README.html

‎.github/workflows/tft.yml‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -157,7 +157,6 @@ jobs:
157157
uses: sclorg/testing-farm-as-github-action@v3
158158
if: contains(needs.prepare_vars.outputs.supported_platforms, matrix.platform)
159159
with:
160-
git_url: https://github.com/linux-system-roles/tft-tests
161160
git_ref: main
162161
pipeline_settings: '{ "type": "tmt-multihost" }'
163162
environment_settings: '{ "provisioning": { "tags": { "BusinessUnit": "system_roles" } } }'
@@ -180,8 +179,7 @@ jobs:
180179
tf_scope: private
181180
api_key: ${{ secrets.TF_API_KEY_RH }}
182181
update_pull_request_status: false
183-
tmt_hardware: '{ "memory": ">= ${{ needs.prepare_vars.outputs.memory }} MB" }'
184-
tmt_plan_filter: "tag:general,storage"
182+
tmt_plan_filter: "tag:playbooks_parallel,storage"
185183

186184
- name: Set final commit status
187185
uses: myrotvorets/set-commit-status-action@master

‎plans/README-plans.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Introduction CI Testing Plans
2+
3+
Linux System Roles CI runs [tmt](https://tmt.readthedocs.io/en/stable/index.html) test plans in [Testing farm](https://docs.testing-farm.io/Testing%20Farm/0.1/index.html) with the [tft.yml](https://github.com/linux-system-roles/storage/blob/main/.github/workflows/tft.yml) GitHub workflow.
4+
5+
The `plans/test_playbooks_parallel.fmf` plan is a test plan that runs test playbooks in parallel on multiple managed nodes.
6+
`plans/test_playbooks_parallel.fmf` is generated centrally from `https://github.com/linux-system-roles/.github/`.
7+
The automation calculates the number of managed nodes to provision with this formula:
8+
9+
```plain
10+
number-of-test-playbooks / 10 + 1
11+
```
12+
13+
The `plans/test_playbooks_parallel.fmf` plan does the following steps:
14+
15+
1. Provisions systems to be used as a control node and as managed nodes.
16+
2. Does the required preparation on systems.
17+
3. For the given role and the given PR, runs the general test from [test.sh](https://github.com/linux-system-roles/tft-tests/blob/main/tests/general/test.sh).
18+
19+
The [tft.yml](https://github.com/linux-system-roles/storage/blob/main/.github/workflows/tft.yml) workflow runs the above plan and uploads the results to our Fedora storage for public access.
20+
This workflow uses Testing Farm's Github Action [Schedule tests on Testing Farm](https://github.com/marketplace/actions/schedule-tests-on-testing-farm).
21+
22+
## Running Tests
23+
24+
You can run tests locally with the `tmt try` cli or remotely in Testing Farm.
25+
26+
### Running Tests Locally
27+
28+
1. Install `tmt` as described in [Installation](https://tmt.readthedocs.io/en/stable/stories/install.html).
29+
2. Change to the role repository directory.
30+
3. Modify `plans/test_playbooks_parallel.fmf` to suit your requirements:
31+
1. Due to [issue #3138](https://github.com/teemtee/tmt/issues/3138), comment out all managed nodes except for one.
32+
2. Optionally modify environment variables to, e.g. run only specified test playbooks by modifying `SYSTEM_ROLES_ONLY_TESTS`.
33+
4. Enter `tmt try -p plans/test_playbooks_parallel <platform>`.
34+
This command identifies the `plans/test_playbooks_parallel.fmf` plan and provisions local VMs, a control node and a managed node.
35+
5. `tmt try` is in development and does not identify tests from URL automatically, so after provisioning the machines, you must type `t`, `p`, `t` from the interactive prompt to identify tests, run preparation steps, and run the tests.
36+
37+
### Running in Testing Farm
38+
39+
1. Install `testing-farm` as described in [Installation](https://gitlab.com/testing-farm/cli/-/blob/main/README.adoc#user-content-installation).
40+
2. Change to the role repository directory.
41+
3. If you want to run tests with edits in your branch, you need to commit and push changes first to some branch.
42+
4. You can uncomment "Inject your ssh public key to test systems" discover step in the plan if you want to troubleshoot tests by SSHing into test systems in Testing Farm.
43+
5. Enter `testing-farm request`.
44+
Edit to your needs.
45+
46+
```bash
47+
$ TESTING_FARM_API_TOKEN=<your_api_token> \
48+
testing-farm request --pipeline-type="tmt-multihost" \
49+
--plan-filter="tag:playbooks_parallel" \
50+
--git-url "https://github.com/<my_user>/storage" \
51+
--git-ref "<my_branch>" \
52+
--compose CentOS-Stream-9 \
53+
-e "SYSTEM_ROLES_ONLY_TESTS=tests_default.yml" \
54+
--no-wait
55+
```

‎plans/test_playbooks_parallel.fmf‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
summary: A general test for a system role
2+
tag: playbooks_parallel
3+
provision:
4+
# TF uses `how: artemis`, and `tmt try`` uses `how: virtual`.
5+
# Hence there is no need to define `how` explicitly.
6+
- name: control-node1
7+
role: control_node
8+
# `connection: system` is required for `how: virtual` to assign VMs a real
9+
# IP making SSH configuration easier.
10+
# This setting is ignored in `artemis`, so we can leave it as is.
11+
connection: system
12+
- name: managed-node1
13+
role: managed_node
14+
connection: system
15+
- name: managed-node2
16+
role: managed_node
17+
connection: system
18+
- name: managed-node3
19+
role: managed_node
20+
connection: system
21+
- name: managed-node4
22+
role: managed_node
23+
connection: system
24+
- name: managed-node5
25+
role: managed_node
26+
connection: system
27+
- name: managed-node6
28+
role: managed_node
29+
connection: system
30+
- name: managed-node7
31+
role: managed_node
32+
connection: system
33+
- name: managed-node8
34+
role: managed_node
35+
connection: system
36+
- name: managed-node9
37+
role: managed_node
38+
connection: system
39+
- name: managed-node10
40+
role: managed_node
41+
connection: system
42+
- name: managed-node11
43+
role: managed_node
44+
connection: system
45+
- name: managed-node12
46+
role: managed_node
47+
connection: system
48+
- name: managed-node13
49+
role: managed_node
50+
connection: system
51+
- name: managed-node14
52+
role: managed_node
53+
connection: system
54+
- name: managed-node15
55+
role: managed_node
56+
connection: system
57+
environment:
58+
ANSIBLE_VER: 2.17
59+
REPO_NAME: storage
60+
PYTHON_VERSION: 3.12
61+
SYSTEM_ROLES_ONLY_TESTS: "" # tests_default.yml
62+
TEST_LOCAL_CHANGES: true
63+
PR_NUM: ""
64+
LINUXSYSTEMROLES_USER: ""
65+
LINUXSYSTEMROLES_DOMAIN: ""
66+
LINUXSYSTEMROLES_SSH_KEY: ""
67+
ARTIFACTS_DIR: ""
68+
ARTIFACTS_URL: ""
69+
LSR_TFT_DEBUG: false
70+
prepare:
71+
- name: Use vault.centos.org repos (CS 7, 8 EOL workaround)
72+
script: |
73+
if grep -q 'CentOS Stream release 8' /etc/redhat-release; then
74+
sed -i '/^mirror/d;s/#\(baseurl=http:\/\/\)mirror/\1vault/' /etc/yum.repos.d/*.repo
75+
fi
76+
if grep -q 'CentOS Linux release 7.9' /etc/redhat-release; then
77+
sed -i '/^mirror/d;s/#\?\(baseurl=http:\/\/\)mirror/\1vault/' /etc/yum.repos.d/*.repo
78+
fi
79+
# Replace with feature: epel: enabled once https://github.com/teemtee/tmt/pull/3128 is merged
80+
- name: Enable epel to install beakerlib
81+
script: |
82+
# CS 10 and Fedora doesn't require epel
83+
if grep -q -e 'CentOS Stream release 10' -e 'Fedora release' /etc/redhat-release; then
84+
exit 0
85+
fi
86+
yum install epel-release yum-utils -y
87+
yum-config-manager --enable epel epel-debuginfo epel-source
88+
discover:
89+
- name: Prepare managed node
90+
how: fmf
91+
where: managed_node
92+
filter: tag:prep_managed_node
93+
url: https://github.com/linux-system-roles/tft-tests
94+
ref: main
95+
- name: Run test playbooks from control_node
96+
how: fmf
97+
where: control_node
98+
filter: tag:test_playbooks
99+
url: https://github.com/linux-system-roles/tft-tests
100+
ref: main
101+
# Uncomment this step for troubleshooting
102+
# This is required because currently testing-farm cli doesn't support running multi-node plans
103+
# You can set ID_RSA_PUB in the environment section above to your public key to distribute it to nodes
104+
# - name: Inject your ssh public key to test systems
105+
# how: fmf
106+
# where: control_node
107+
# filter: tag:reserve_system
108+
# url: https://github.com/linux-system-roles/tft-tests
109+
# ref: main
110+
execute:
111+
how: tmt

0 commit comments

Comments
 (0)