Skip to content

Task ‐ Cross‐Platform Test Strategy

Jump to bottom Edit New page
H. Joe Lee edited this page Oct 9, 2025 · 4 revisions

Objective:

Create a comprehensive test plan to validate HDF5 functionality, performance, and user experience across Windows, macOS, and Linux. A key component of this task is to formally document the justification and strategic reasoning for supporting and testing each specific platform.

Deliverables:

The testing team is responsible for producing the following documents:

1. Platform Justification and Risk Analysis

This document will serve as the foundation for the test plan. For each target operating system (Windows, macOS, and Linux), you must provide:

  • User Base Case: Articulate the primary user base and opportunities for each platform. Why is it strategically crucial for HDF5 to function flawlessly on this operating system?
  • Technical Risk Assessment: Identify the unique technical risks and potential failure points inherent to each OS. Your analysis must detail at least two platform-specific examples, such as:
    • Windows: File system path syntax (\), registry dependencies, User Account Control (UAC) permissions.
    • macOS: Application sandboxing, Gatekeeper notarization, differences between Intel and Apple Silicon architectures.
    • Linux: Package management fragmentation (e.g., .deb vs. .rpm), variations in software environments, and system library dependencies (e.g., glibc versions).
  • Testing Rationale: Conclude with a summary explaining why a dedicated testing effort on each OS is critical to mitigate the identified risks and ensure we meet user expectations.

2. Comprehensive Test Plan

Based on your justification document, create a detailed test plan that includes:

  • Environment Specification: Define the exact versions of each OS and any required hardware configurations (e.g., Windows 11, Ubuntu 24.04 LTS, macOS Sonoma on M2).
  • Scope and Scenarios: Outline the test cases, distinguishing between universal core functionality and tests for platform-specific features.
  • Methodology: Define the types of testing to be performed (functional, performance, UI/UX, compatibility, etc.) and the approach for each.
  • Automation Strategy: Propose which test scenarios are candidates for automation and the tools you will use to implement them consistently across all three platforms.
  • Testing Results Dashboard: Describe how testing reporting will work and how it will be standardized.

HDF5 Cross-Platform Testing Strategy

Executive Summary

This document outlines the comprehensive cross-platform testing strategy for HDF5, based on the GitHub Actions CI/CD infrastructure defined in .github/workflows/*.yml and the test matrix documented in WATCHME.md. The strategy ensures HDF5 builds and functions correctly across diverse architectures, operating systems, compilers, and configurations.

Testing Dimensions

1. Architecture Coverage

HDF5 is tested across 7 major CPU architectures to ensure portability:

Architecture Platform Workflow Purpose
i386 Alpine Linux i386-alp.yml, i386-alp-f.yml 32-bit x86 compatibility
amd64 Multiple r.yml Standard 64-bit x86
arm64 Linux, macOS, FreeBSD arm64.yml, arm64-lin.yml, arm64-fbsd-f-j.yml Apple Silicon & ARM64 servers
armv7 Linux armv7-lin.yml 32-bit ARM (IoT, embedded)
riscv64 Linux riscv64-lin.yml, riscv64-lin-sh.yml RISC-V emerging architecture
ppc64le Linux ppc64le-lin.yml IBM POWER (big data, HPC)
s390x Linux s390x-lin.yml, s390x-lin-f.yml, s390x-lin-mpich.yml IBM Z mainframe

Strategy: Each architecture is tested with at least one basic configuration. Critical architectures (amd64, arm64) receive extended testing with multiple compiler and feature combinations.

2. Operating System Coverage

BSD Family

OS Version Workflow Key Features
FreeBSD 14.3 fbsd.yml, fbsd-f-j.yml, fbsd-lf-j.yml, fbsd-ompi.yml Java, Fortran, OpenMPI 5, LFortran
NetBSD 9.3, latest nbsd.yml, nbsd-9.3.yml, nbsd-f.yml, nbsd-f-j.yml, nbsd-f-sh.yml Fortran, Java, shared libs

Testing Approach: Uses cross-platform-actions/action@master to run BSD VMs on Ubuntu hosts. Tests core functionality, language bindings, and MPI support.

macOS (Darwin)

Version Codename Workflow Key Features
macOS 14 Sonoma mac14-*.yml Xcode 16, Clang 19, Flang
macOS 15 Sequoia mac15.yml, mac15-xcode.yml Latest OS testing
iOS - ios.yml Mobile platform

Strategy:

  • Test with both Xcode and Homebrew compilers
  • Verify MPI support with OpenMPI and MPICH
  • Test Python integration via Spack (h5py)
  • Validate Java bindings and thread-safety

Linux Distributions

Distribution Workflow Purpose
Ubuntu Most workflows Primary CI platform (latest, 24.04-arm)
SUSE sus.yml, sus_t.yml Enterprise Linux validation

Strategy: Ubuntu serves as the primary testing platform due to GitHub Actions support. SUSE validates enterprise distributions.

Solaris

OS Workflow Notes
Solaris sol.yml Unix System V compatibility

Windows

Environment Workflow Compiler
Native Windows win-ninja-icx*.yml Intel ICX/IFX
MSYS2 msys2.yml GCC on Windows
Cygwin cyg-ninja.yml, cyg-ninja-sh-ompi.yml POSIX on Windows

Strategy:

  • Native Windows with Intel compilers for production builds
  • MSYS2 for GCC compatibility
  • Cygwin for POSIX API validation
  • Test parallel builds with Intel MPI

3. Build System & Tools

Build System Workflows Use Cases
CMake + Ninja *-ninja-*.yml (30+ workflows) Fast parallel builds on all platforms
CMake + Make Default in most workflows Traditional builds
Spack lin-spack-f-ompi.yml, mac-spack-h5py.yml Package manager integration
Xcode mac-xcode.yml, mac*-xcode*.yml macOS native builds

Strategy: Ninja is preferred for speed. Spack validates package ecosystem integration. Xcode ensures macOS native toolchain compatibility.

4. Compiler Coverage

C Compilers

Compiler Workflows Platforms Notes
GCC Default on Linux/BSD All Linux, BSD Minimum version tested via lin-gcc_min.yml
GCC ARM lin-gcc_arm64.yml, lin-gcc_arm64-cpp.yml ARM64 Cross-compilation
Clang/LLVM mac15-ninja-clang*.yml, lin-clang-s_l.yml macOS, Linux Versions 15, 17, 19
Intel ICX lin-icx*.yml, win-ninja-icx*.yml Linux, Windows oneAPI compiler
NVIDIA NVHPC lin-nvhpc*.yml, lin-ninja-nvhpc*.yml Linux HPC optimizations
AOCC aocc.yml, lin-aocc-ompi*.yml Linux AMD optimizing compiler
Android NDK lin-ndk.yml Android/Linux Mobile platform

Fortran Compilers

Compiler Workflows Notes
gfortran Default in *-f.yml workflows Standard Fortran compiler
Intel IFX win-ninja-icx-f*.yml Modern Intel Fortran
LFortran fbsd-lf-j.yml Modern LLVM-based Fortran
Flang mac14-ninja-clang19-fl-d-f-mpich-sf.yml LLVM Fortran frontend
NVIDIA Fortran lin-nvhpc-f*.yml NVHPC suite

Strategy:

  • Test with minimum supported GCC version
  • Validate modern compilers (ICX, NVHPC, AOCC)
  • Ensure Fortran API compatibility across vendors
  • Use leak sanitizers with Clang (lin-clang-s_l.yml)

5. Parallel Computing (MPI)

MPI Implementation Workflows Platforms Configuration
OpenMPI ompi.yml, *-ompi*.yml (15+ workflows) Linux, macOS, Cygwin, FreeBSD Versions 4.x and 5.x
MPICH *-mpich*.yml (8+ workflows) Linux, macOS Standard MPI implementation
Intel MPI win-ninja-icx-*-p*.yml, lin-icx-f-p-sf.yml Windows, Linux Part of oneAPI

Testing Strategy:

  • Configure with MPIEXEC_MAX_NUMPROCS=2 to simulate parallel execution
  • Test subfiling VFD with MPI (-p-sf workflows)
  • Validate both static and shared MPI builds
  • Test MPI I/O operations

Key Test Workflows:

  • .github/workflows/build_openmpi_source.yml - Build OpenMPI from source
  • .github/workflows/build_mpich_source.yml - Build MPICH from source

6. Feature Matrix

Language Bindings

Language Workflows Test Coverage
C All workflows Core API
C++ *-cpp*.yml, default in many C++ API, STL integration
Fortran *-f.yml (30+ workflows) F90/F2003/F2008 APIs
Java *-j.yml (10+ workflows) JNI bindings
Python mac-spack-h5py.yml h5py integration
Julia lin-jl.yml Julia bindings

Core Features

Feature Flag Workflows Purpose
High-Level Library HDF5_BUILD_HL_LIB Most workflows HL APIs (tables, images, etc.)
Parallel I/O HDF5_ENABLE_PARALLEL *-p*.yml, *-ompi*.yml, *-mpich*.yml MPI-based parallel I/O
Thread Safety HDF5_ENABLE_THREADSAFE mac-ninja-clang-j-ompi-ts.yml Multi-threaded applications
Shared Libraries BUILD_SHARED_LIBS *-sh*.yml Dynamic linking
Debug Builds CMAKE_BUILD_TYPE=Debug d.yml, *-d-*.yml Debugging support

Virtual File Drivers (VFDs)

VFD Workflows Description
Subfiling *-sf.yml, vfd-subfiling.yml Distributed file I/O for HPC
ROS3 mac15-sh-s3.yml, vfd-ros3.yml Read-only S3 access
General VFD vfd.yml, vfd-main.yml All VFD tests

Virtual Object Layer (VOL)

VOL Connector Workflow Description
REST vol_rest.yml RESTful API access
Async vol_async.yml Asynchronous I/O
ADIOS2 vol_adios2.yml ADIOS2 integration
Cache vol_cache.yml Caching layer
Log vol_log.yml I/O logging
Pass-through vol_ext_passthru.yml External pass-through

Testing Approach: Scheduled daily at 06:00 CDT via vol.yml orchestrator.

7. Build Configurations

Configuration Workflows Purpose
Release Default in most workflows Production builds
RelWithDebInfo lin-icx-f-p-sf.yml Optimized with debug symbols
Debug d.yml, *-d-*.yml Full debugging
All Features OFF mac15-off.yml Minimal build validation
Warnings as Errors Most workflows with HDF5_ENABLE_ALL_WARNINGS Code quality enforcement

8. Compression & I/O Libraries

Library Configuration Workflows Notes
Zlib Often enabled by default Most workflows Compression
SZIP HDF5_ENABLE_SZIP_SUPPORT=OFF Most workflows Disabled due to licensing
AECLIB Installed via package managers BSD, macOS workflows Alternative to SZIP
MKL MKL_ROOT=/opt/intel/oneapi/mkl/latest Intel workflows Math kernel library

Testing Methodology

1. Continuous Integration Triggers

on:
  push:
    branches:
      - develop

Strategy: Most workflows trigger on every push to develop branch for immediate feedback.

Special Cases:

  • workflow_dispatch: Manual trigger for on-demand testing (e.g., arm64-lin.yml)
  • schedule: Daily runs for extensive tests (e.g., vol.yml at 11:00 UTC)
  • workflow_call: Reusable workflows (e.g., vfd-subfiling.yml, abi_c.yml)

2. Test Execution Pattern

Standard workflow pattern:

  1. Setup: Install dependencies (compilers, MPI, libraries)
  2. Configure: Run CMake with feature flags
  3. Build: Compile HDF5 library and tests
  4. Test: Execute CTest suite
  5. Upload: Artifact preservation (test costs, results)

Example from lin-icx-f-p-sf.yml:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE="RelWithDebInfo" \
  -DHDF5_BUILD_FORTRAN=ON \
  -DHDF5_ENABLE_PARALLEL=ON \
  -DHDF5_ENABLE_SUBFILING_VFD:BOOL=ON \
  -DCMAKE_C_COMPILER=mpiicx \
  -DCMAKE_Fortran_COMPILER=mpiifx

cmake --build build
ctest --test-dir build --output-on-failure

3. Matrix Strategy

Some workflows use job matrices for combinatorial testing:

strategy:
  matrix:
    os_name: ["Ubuntu"]
    mpi_lib: ["OpenMPI"]

Benefits: Tests multiple configurations without duplicating workflow files.

4. Cross-Platform Actions

BSD testing uses VM emulation:

- uses: cross-platform-actions/action@master
  with:
    operating_system: freebsd
    architecture: x86-64
    version: '14.3'

5. CDash Reporting

Many workflows integrate with CDash for test result aggregation:

-DCTEST_DROP_SITE_INIT:STRING="my.cdash.org"
-DSITE:STRING="ubu-24.arm64"
-DBUILDNAME:STRING="-ex"
ctest -D Experimental

Specialized Testing

1. ABI Compatibility (abi.yml, abi_c.yml, abi-report.yml)

  • Tests API/ABI compatibility between HDF5 versions
  • Compares versions 1.14.3 vs 1.15.4
  • Ensures backward compatibility

2. Binary Distribution Testing (bintest.yml)

  • Validates pre-built binary packages
  • Tests installation and basic functionality

3. Release Workflow (release.yml)

  • Automated release process
  • Tarball generation and validation

4. Documentation (docs.yml)

  • Builds and validates documentation
  • Ensures doc generation succeeds

5. Code Quality

  • Formatting: clang-format-check.yml, clang-format-fix.yml
  • Link Checking: markdown-link-check.yml
  • Static Analysis: analysis.yml

6. Interoperability (WATCHME.md Extra section)

  • HDF-EOS5: Tests compatibility with HDF-EOS5 library
  • netCDF: Validates netCDF integration

Test Infrastructure Management

Reusable Workflows

Workflow Purpose Called By
build_openmpi_source.yml Build OpenMPI from source MPI testing workflows
build_mpich_source.yml Build MPICH from source MPI testing workflows
vfd-subfiling.yml Test subfiling VFD VFD main workflow
abi_c.yml ABI compatibility checking abi.yml

Workflow Orchestration

  • call-workflows.yml: Master workflow for coordinated testing
  • arm-main.yml: ARM architecture test orchestrator
  • vfd-main.yml: VFD test orchestrator

Coverage Gaps & Recommendations

Current Coverage Strengths

✅ Excellent architecture diversity (7 CPU families) ✅ Comprehensive OS coverage (BSD, Linux, macOS, Windows, Solaris) ✅ Multiple compiler vendors (GCC, Clang, Intel, NVIDIA, AMD, LFortran) ✅ Strong MPI testing (OpenMPI, MPICH, Intel MPI) ✅ Language binding validation (C, C++, Fortran, Java, Python, Julia)

Potential Enhancements

⚠️ macOS Coverage: Some commented-out MacOS matrix entries in vfd-subfiling.yml ⚠️ MPICH on macOS: Currently disabled in VFD testing ⚠️ Windows MPI: Limited to Intel MPI; could add MS-MPI or MPICH ⚠️ Android: NDK tested only on Linux; actual Android runtime not tested ⚠️ Containerization: Consider Docker-based testing for reproducibility

Performance Considerations

Test Optimization Strategies

  1. Parallel Builds: Use Ninja generator for faster compilation
  2. Selective Testing: VFD workflows run targeted test subsets via -R regex
  3. Artifact Caching: Upload only CTestCostData.txt to minimize storage
  4. Scheduled Runs: Expensive tests (VOL connectors) run daily, not per-commit
  5. Fail-Fast Control: Many workflows use fail-fast: false to collect all results

Resource Management

  • Runner Selection: Use architecture-specific runners (e.g., ubuntu-24.04-arm)
  • Timeout Controls: Implicit via CTest timeout settings
  • Dependency Caching: Package managers cache installations (apt, brew, pkg)

Maintenance & Evolution

Naming Convention

Workflows follow a consistent naming pattern:

  • {os}-{build}-{compiler}-{features}.yml
  • Example: lin-ninja-nvhpc-f-p-sf.yml = Linux + Ninja + NVHPC + Fortran + Parallel + Subfiling

Version Pinning

  • Actions: Pinned to specific versions (e.g., actions/[email protected])
  • Compilers: Specific versions tested (e.g., Intel 2025.0, Clang 15/17/19)
  • OS Versions: Explicit versions (e.g., FreeBSD 14.3, NetBSD 9.3)

Badge Status Tracking

WATCHME.md provides real-time CI status via GitHub Actions badges, organized by:

  • Architecture
  • Operating System
  • Build System
  • Compiler
  • Configuration
  • Languages
  • Parallel Computing
  • Thread Safety
  • VFDs

Conclusion

HDF5's cross-platform testing strategy represents one of the most comprehensive CI/CD infrastructures in the scientific computing ecosystem. With 100+ GitHub Actions workflows covering 7 CPU architectures, 5 OS families, 10+ compiler toolchains, and extensive feature combinations, the strategy ensures HDF5's reliability across the entire spectrum of modern computing platforms.

The strategy balances breadth (maximum platform coverage) with depth (thorough feature testing on key platforms), while optimizing for CI resource efficiency through selective test execution and scheduled runs for expensive test suites.

Document Version: 1.0 Based on: .github/workflows/*.yml (100+ workflow files) and WATCHME.md Last Updated: 2025-10-07

Add a custom footer
Clone this wiki locally