help improving testing

[n0xena]

log.txt
results.txt

So I just finished ZTS of 2.4.0-rc1 on Arch with standard 6.16.2:

Results Summary
PASS     1854
FAIL      15
SKIP      14
KILLED     2

Running Time:   08:12:49
Percent passed: 98.4%
Log directory:  /var/tmp/test_results/20250823T160421

Tests with results other than PASS that are expected:
    FAIL casenorm/mixed_formd_delete (https://github.com/openzfs/zfs/issues/7633)
    FAIL casenorm/mixed_formd_lookup (https://github.com/openzfs/zfs/issues/7633)
    FAIL casenorm/mixed_formd_lookup_ci (https://github.com/openzfs/zfs/issues/7633)
    FAIL casenorm/mixed_none_lookup_ci (https://github.com/openzfs/zfs/issues/7633)
    FAIL casenorm/sensitive_formd_delete (https://github.com/openzfs/zfs/issues/7633)
    FAIL casenorm/sensitive_formd_lookup (https://github.com/openzfs/zfs/issues/7633)
    SKIP cli_root/zfs_unshare/zfs_unshare_006_pos (Not applicable)
    FAIL cli_root/zpool_import/import_rewind_device_replaced (Arbitrary pool rewind is not guaranteed)
    SKIP cli_root/zpool_import/zpool_import_missing_003_pos (https://github.com/openzfs/zfs/issues/6839)
    FAIL fault/auto_spare_multiple (https://github.com/openzfs/zfs/issues/11889)
    FAIL fault/auto_spare_shared (https://github.com/openzfs/zfs/issues/11889)
    SKIP io/io_uring (io_uring support required)
    FAIL mmap/mmap_sync_001_pos (Known issue)
    SKIP pool_checkpoint/checkpoint_discard_busy (https://github.com/openzfs/zfs/issues/12053)
    FAIL refreserv/refreserv_004_pos (Known issue)
    SKIP removal/removal_with_zdb (Known issue)
    SKIP rsend/rsend_008_pos (https://github.com/openzfs/zfs/issues/6066)
    FAIL vdev_zaps/vdev_zaps_007_pos (Known issue)

Tests with result of PASS that are unexpected:

Tests with results other than PASS that are unexpected:
    FAIL cli_root/zfs_share/zfs_share_005_pos (expected PASS)
    KILLED cli_root/zpool_prefetch/zpool_prefetch_001_pos (expected PASS)
    KILLED cp_files/cp_stress (expected PASS)
    FAIL fault/auto_spare_001_pos (expected PASS)
    SKIP mmap/mmap_libaio_001_pos (expected PASS)
    FAIL slog/slog_replay_fs_001 (expected PASS)

Overall the ZTS readme lacks a lot of info about the required dependencies - maybe this can be improved in a distro-independed manner like "for proper testing you need these packages: X, Y, Z, ...".
As for the libaio skip: The archzfs build is designed to use clean-chroot-manager. The mmap_libaio binary requires libaio to be present in the build environment - which archzfs doesn't provide. This can be fixed by building the utils package manually - but I just didn't bother. Last time the related tests ran fine. So I take the SKIP as "expected as build environment issue".
Can anyone share some insides into what I can do to clear the few failed tests?

I opened this as discussion rather than issue as I see this as an issue on my end rather than of the project. I'd just like to provide some feedback.

[behlendorf]

@n0xena thanks for testing out 2.4.0-rc1 and posting your results.

Overall the ZTS readme lacks a lot of info about the required dependencies - maybe this can be improved in a distro-independed manner like "for proper testing you need these packages: X, Y, Z, ...".

It's hard to find, but the full list of package dependencies does exist in the repository. I'm all for extracting from those lists the extra packages needed to run ZTS and adding them to the README. The tricky bit is usually keeping this information current. Please feel free to open a PR.

For Arch in particular it would be nice to revisit this PR to add it to our CI, #16816. If you're interested, the bulk of the remaining work was to make it an optional builder, see #16816 (comment). The Arch kernel can move too fast for us at times, but having the ability to easily run the ZTS on Arch would help us detect and address any failures like you saw.

[n0xena]

It's hard to find, but the full list of package dependencies does exist in the repository. I'm all for extracting from those lists the extra packages needed to run ZTS and adding them to the README. The tricky bit is usually keeping this information current. Please feel free to open a PR.

I think it would help a lot to mention this in the ZTS readme right after the FreeBSD line.
As for Arch, there's some manual work to do for Samba, which is explained in the wiki: https://wiki.archlinux.org/title/Samba#Installation.
Someone willing to run ZTS should be able to figure this out. I noticed this because, upon rebooting, I saw that SMBd was failing to start, so I looked up the error.
Given the complexity of the script, it's clear that it's not as easy to include all of that information in the readme itself. Perhaps an additional page could be added to provide more details.

For Arch in particular it would be nice to revisit this PR to add it to our CI, #16816. If you're interested, the bulk of the remaining work was to make it an optional builder, see #16816 (comment). The Arch kernel can move too fast for us at times, but having the ability to easily run the ZTS on Arch would help us detect and address any failures like you saw.

I'm happy to assist. After evaluating and testing pretty much every solution (hardware RAID, fake RAID, Storage Spaces, BtrFS, LVM, and classic MD RAID), I found that ZFS best fits my needs. As for Arch: I switched to it because a friend of mine used it, and I just stuck with it. I used to run OpenSuSE—it was my first "Linux"—and I still use it on my servers. However, as I mainly game on my system, even the most recent versions aren't that great or well supported.
As one can see from my ArchZFS fork, I try to give back to the community from which I benefit so much. Why not spend some hours running ZTS on the cutting edge?
I'll look into it and see if I can provide some Arch-related stuff to maybe piece something together. I would love to support this amazing project in any way.