SRE-3150 ci: enable HW Tests on el9.6 #17327

grom72 · 2025-12-30T15:19:09Z

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]> Signed-off-by: Samir Raval <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Skip-build-el9-gcc: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Skip-build-el9-gcc: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]>

This reverts commit e3da5b4. Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw

Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Doc-only: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]>

Let's try to run fix with original configuration to confirm that problem still exists. Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Revert "Run Tests without fix to collect more data" This reverts commit 2bae9ce. Signed-off-by: Tomasz Gromadzki <[email protected]>

clamav must be removed as required dependency is not avaialble: "nothing provides libcrypto.so.3(OPENSSL_3.4.0)(64bit)" Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

This reverts commit e2ba246. Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

daosbuild3 · 2026-01-02T13:09:55Z

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

daosbuild3 · 2026-01-02T13:14:21Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

daosbuild3 · 2026-01-02T13:19:08Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

daosbuild3 · 2026-01-02T13:19:28Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/16/execution/node/640/log

daosbuild3 · 2026-01-02T13:19:30Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

daosbuild3 · 2026-01-02T13:19:52Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/16/display/redirect

daosbuild3 · 2026-01-02T13:39:53Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

daosbuild3 · 2026-01-02T13:40:10Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

daosbuild3 · 2026-01-02T13:40:47Z

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

Cancel-prev-build: false Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-8.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <[email protected]>

daosbuild3 · 2026-01-02T13:52:08Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

daosbuild3 · 2026-01-02T13:52:25Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/17/display/redirect

daosbuild3 · 2026-01-02T13:59:49Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/17/execution/node/614/log

daosbuild3 · 2026-01-02T14:23:23Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

daosbuild3 · 2026-01-02T14:27:45Z

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

daosbuild3 · 2026-01-02T14:35:27Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

daosbuild3 · 2026-01-02T14:48:50Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

daosbuild3 · 2026-01-02T14:48:57Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/18/display/redirect

daosbuild3 · 2026-01-02T15:18:21Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/18/execution/node/722/log

daosbuild3 · 2026-01-02T19:49:44Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/18/execution/node/707/log

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-8.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <[email protected]>

daosbuild3 · 2026-01-02T22:03:41Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/19/execution/node/696/log

daosbuild3 · 2026-01-02T22:34:20Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/19/execution/node/856/log

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-8.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <[email protected]>

daosbuild3 · 2026-01-02T23:23:00Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/20/display/redirect

daosbuild3 · 2026-01-03T00:09:52Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/21/execution/node/670/log

daosbuild3 · 2026-01-03T01:34:48Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/19/execution/node/681/log

daosbuild3 · 2026-01-03T04:10:11Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/21/execution/node/660/log

JohnMalmberg · 2026-01-09T14:35:50Z

ci/provisioning/post_provision_config_nodes_EL.sh

    dnf -y reinstall sssd-common
+    # Seems to be required until https://daosio.atlassian.net/browse/DAOS-18358
+    # is fixed.
+    dnf -y remove clamav-lib


This is because el-9.6 is out of standard support and clamav-lib is no longer supports the out of support release. It should work with the current el-9.7 release.

ravalsam and others added 30 commits December 30, 2025 15:07

SRE-3150 Test: This is Debug PR for NVMe issue

d5cb8e0

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Reverting it back to original OS.

6c78311

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Comment node prep NVMe check.

fdb99d2

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Update test_main_prep_node.sh

9f60be9

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]> Signed-off-by: Samir Raval <[email protected]>

Testing changes.

e4a7c38

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Break tests repetition after first failure

3f3b111

Skip-build-el9-gcc: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Sanity check on el9.5

19a6a3c

Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Skip leap 15 build

8aed634

Signed-off-by: Tomasz Gromadzki <[email protected]>

Revert "Sanity check on el9.5"

481de85

This reverts commit e3da5b4. Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Let's try 9.5 again

48bc314

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw

Add parameter to controll HW stages image version

74ebd21

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw

Stop tests as soon as fault is detected

455f4d5

Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

nvme drives setup script

fde1309

Doc-only: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Increase max repetition number for cluster sanity check

5b7b932

Signed-off-by: Tomasz Gromadzki <[email protected]>

Update script to enable two devices per numa node

a77ad8b

Signed-off-by: Tomasz Gromadzki <[email protected]>

Speed up test by les repetitions

0ad5d6e

Signed-off-by: Tomasz Gromadzki <[email protected]>

Avoid error in case of unbinding failure

504ad56

Signed-off-by: Tomasz Gromadzki <[email protected]>

grom72 force-pushed the grom72/SRE-3150 branch from e5972e0 to cd8846f Compare January 2, 2026 13:55

JohnMalmberg reviewed Jan 9, 2026

View reviewed changes

SRE-3150 ci: enable HW Tests on el9.6 #17327

Are you sure you want to change the base?

SRE-3150 ci: enable HW Tests on el9.6 #17327

Conversation

grom72 commented Dec 30, 2025

Steps for the author:

After all prior steps are complete:

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 3, 2026

Uh oh!

daosbuild3 commented Jan 3, 2026

Uh oh!

daosbuild3 commented Jan 3, 2026

Uh oh!

JohnMalmberg Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants