HDF5 output-correctness tests, driver unit tests, build-option detection, Codecov workflow - #157
Merged
Conversation
jeanbez
changed the title
jeanbez
force-pushed
the
wave-c-prep-tests
branch
from
June 5, 2026 02:04
a1e6a66 to
4c4a638
Compare
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
jeanbez
marked this pull request as ready for review
June 29, 2026 01:51
jeanbez
added
enhancement
Before refactoring commons/ and h5bench_patterns/ in Wave C we need tests
that catch silent data corruption, not just silent-exit-zero runs. The
existing pytest suite only asserts that a sample config file exists on
disk; the VALIDATE flag only greps stdout for a mode string. This patch
adds a real content-validation layer.
New structure
tests/helpers/h5_assert.py — h5py-based assertion library:
list_timestep_groups, dataset_sha256, assert_timestep_count,
assert_contig_datasets_present, assert_dataset_shape,
assert_dataset_dtype, assert_value_in_range,
assert_unlimited_axis, assert_id_1_1d_pattern,
assert_id_2_1d_write_pattern, assert_datasets_equal.
tests/fixtures/tiny-*.json — 1024-particle, 2-rank, 1-2-timestep
configs. These are test-owned (do not touch the production
samples/) and run in a few seconds each.
tests/test_output_validation.py — exercises write / append /
overwrite / write-unlimited patterns; checks group + dataset
presence, shape, dtype, value ranges, the deterministic id_1 /
id_2 patterns on the 1D contig write, H5S_UNLIMITED axis
survival, and the "overwrite actually overwrites id_2" invariant.
tests/test_read_roundtrip.py — closes the h5bench_read gap: runs
write-then-read through the driver and verifies the file survives
the read step with the expected structure and id_1 pattern.
tests/requirements.txt — h5py, numpy.
Infrastructure changes
tests/conftest.py — extracted the VOL-path injection into a helper
and glob the new tests/fixtures/ alongside the legacy samples/ so
both feed through identical pre-processing.
tests/CMakeLists.txt — register h5bench-output-validation and
h5bench-read-roundtrip with ctest, mirroring the existing
h5bench-sync convention.
A few subtleties worth noting:
* mpi.configuration is respected verbatim by the driver, so the
fixtures include "-np 2" explicitly rather than relying on
mpi.ranks (which is only consulted when configuration is empty).
* h5bench_write overwrites X_DIM/Y_DIM/Z_DIM at runtime with
X_RAND/Y_RAND/Z_RAND (191/1009/3701), so the value-range assertion
uses those; this is an odd but real part of the current contract
Wave C must preserve.
* The append step does not grow the file beyond 2 * dim_1: with 2
ranks writing dim_1 particles via write-unlimited first, the file
already has extent 2 * dim_1, so append semantically overwrites the
second half. The test asserts exactly that invariant.
Verified locally (HDF5 2.1.1, macOS / AppleClang): 5 new tests pass; 24
existing sync tests still pass. No behaviour change for CI until this
lands as its own PR and tests/requirements.txt gets installed in the
workflows - intentionally deferred.
Extends the Tier 1 scaffolding with fixtures that drive the three
remaining MEM_PATTERN x FILE_PATTERN combinations of h5bench_write,
2D + 3D contig variants, and the previously-untested
h5bench_write_var_normal_dist binary.
New fixtures (tests/fixtures/)
tiny-1d-contig-interleaved.json -> CONTIG mem / INTERLEAVED file
tiny-1d-interleaved-contig.json -> INTERLEAVED mem / CONTIG file
tiny-1d-interleaved-interleaved.json -> INTERLEAVED mem / INTERLEAVED file
tiny-2d-write.json -> NUM_DIMS=2, 32x16 per rank
tiny-3d-write.json -> NUM_DIMS=3, 8x8x8 per rank
tiny-write-var-normal-dist.json -> write_var_normal_dist, stdev 128
New / extended tests
test_output_validation.py -> + 3 compound/interleaved tests,
+ 2D and 3D structural tests
test_write_var_normal_dist.py -> first real pytest for the
h5bench_write_var_normal_dist
binary (the existing
test_sync_h5bench_normal_dist.py
has two latent bugs: wrong
binary name and non-existent
sample filename, so it has
never actually run)
h5_assert.py gains assert_compound_dataset_present() for the
compound-type output variant (single /Timestep_N/particles dataset with
an 8-field compound HDF5 type).
CMakeLists.txt registers h5bench-write-var-normal-dist with ctest.
Coverage (measured with gcovr + coverage.py against an instrumented
local build, 35 tests run):
C lines: 55.7% -> 64.0% (+8.3)
C functions: 59.7% -> 69.5% (+9.8)
C branches: 39.2% -> 45.3% (+6.1)
write_normal_dist: 0% -> 51%
Python driver: 37% -> 37% (flat - expected; these tests all
exercise the run_pattern dispatcher)
The remaining C gaps live behind USE_COMPRESS, HAVE_SUBFILING, and the
ASYNC VOL code paths; Python gains require tests for the non-pattern
benchmarks (exerciser / metadata / amrex / openpmd / e3sm / macsio) and
negative driver paths (validate_json failures, missing binary).
Adds tests/test_driver_unit.py with 30 pytest cases for the parts of
src/h5bench.py that don't need a benchmark binary:
* validate_json: all 5 required keys, each missing-key path exits
with EX_DATAERR, extra keys tolerated.
* prepare_parallel: mpirun + mpiexec + srun + explicit-configuration
+ unknown-command branches.
* is_available: shutil.which hit in PATH vs. miss.
* check_parallel: plain shell vs. $SHELL containing mpirun|mpiexec|
srun (which exits with EX_USAGE) vs. $SHELL unset.
* prepare_vol / enable_vol / disable_vol / reset_vol: None input,
library + preload + path all present, connector key optional,
disable when connector unset, reset clears HDF5_PLUGIN_PATH +
HDF5_VOL_CONNECTOR + ABT_THREAD_STACKSIZE.
* check_for_hdf5_error: clean stderr returns False, HDF5 error
banner triggers sys.exit(EX_IOERR).
* run() negative paths: missing config file -> EX_NOINPUT, malformed
JSON -> EX_NOINPUT.
CMakeLists.txt registers h5bench-driver-unit with ctest.
Coverage (measured with exerciser + metadata binaries built; 67 tests):
Python src/h5bench.py 37% -> 54% (+17)
C lines 64.0% -> 64.6% (+0.6)
C functions 69.5% -> 72.5% (+3.0)
The remaining Python gap (46%) is almost entirely the optional-benchmark
dispatchers (run_amrex / run_openpmd / run_e3sm / run_macsio) which need
those submodules built, main()/argparse entry point, Lustre detection,
and ASYNC-mode sub-branches.
30 new unit tests run in ~0.05s total - no MPI, no HDF5, no subprocess.
Introduces a build-time option introspection mechanism and rewires the
optional benchmark tests (AMReX, OpenPMD, E3SM, MACSio, ASYNC VOL,
exerciser, metadata) to key off it. Previous behaviour: tests globbed
for a binary file and skipped on "not found" - which hid the difference
between "option was off" (expected) and "option was on but the binary
failed to build" (bug).
src/h5bench_configuration.py.in gains an __options__ dict populated by
CMake's configure_file substitution at configure time. Every h5bench
flag + WITH_ASYNC_VOL is surfaced as a real Python bool:
__options__ = {
"H5BENCH_METADATA": _flag("@H5BENCH_METADATA@"),
"H5BENCH_EXERCISER": _flag("@H5BENCH_EXERCISER@"),
"H5BENCH_AMREX": _flag("@H5BENCH_AMREX@"),
"H5BENCH_OPENPMD": _flag("@H5BENCH_OPENPMD@"),
"H5BENCH_E3SM": _flag("@H5BENCH_E3SM@"),
"H5BENCH_MACSIO": _flag("@H5BENCH_MACSIO@"),
"WITH_ASYNC_VOL": _flag("@WITH_ASYNC_VOL@"),
}
tests/helpers/build_config.py exposes a requires_<thing> pytest.mark.
skipif for each entry. The reason string always includes the flag name
and state, so `pytest -v` output is self-explanatory, e.g.:
SKIPPED (H5BENCH_E3SM=OFF at build time (E3SM-IO benchmark))
If h5bench_configuration can't be imported (running pytest out of tree
without cmake), the helper falls back to "everything off" so tests
skip cleanly instead of erroring.
Refactored tests now use the markers and upgrade binary-missing from a
silent skip to a hard assert: if H5BENCH_AMREX=ON but h5bench_amrex_sync
is missing, that's a build regression and the test fails loudly rather
than hiding.
test_sync_amrex.py -> @requires_amrex
test_async_amrex.py -> @requires_amrex + @requires_async_vol
test_sync_openpmd.py -> @requires_openpmd
test_async_h5bench.py -> @requires_async_vol
test_sync_exerciser.py -> @requires_exerciser
test_sync_metadata.py -> @requires_metadata
New tests that didn't exist before:
test_sync_e3sm.py -> @requires_e3sm
test_sync_macsio.py -> @requires_macsio
(CMake registers both with ctest, gated on the matching option.)
Verified locally against an EXERCISER+METADATA-only build: 66 pattern /
unit / metadata tests pass, 23 optional-dep tests skip with explicit
per-flag reasons, and one pre-existing macOS-specific h5bench_exerciser
SIGTRAP on sync-exerciser.json - unrelated to this change.
New .github/workflows/coverage.yml collects line + branch coverage
for the h5bench C benchmarks and the Python driver, then uploads both
reports to Codecov. Single HDF5 version (1.14.0) with every optional
benchmark and VOL-ASYNC enabled, so the one workflow reflects the
whole coverage surface rather than duplicating instrumentation across
nine per-HDF5-version workflows.
Triggers
* pull_request to develop or master - PR-level coverage signal
* push to master - baseline after merge
* workflow_dispatch - manual re-run
* schedule, Mondays 06:00 UTC - weekly drift check
Build + test flow
* Installs coverage tooling (gcovr, coverage.py, h5py, numpy) into
an isolated venv so PEP 668 base images (Ubuntu 24.04+) don't block
pip.
* Clones + builds Argobots and VOL-ASYNC (v1.7) to /opt, same
recipe the production workflow uses.
* One h5bench CMake configure with -DH5BENCH_ALL=ON +
-DWITH_ASYNC_VOL:BOOL=ON + --coverage on CFLAGS/CXXFLAGS and
EXE/SHARED linker flags so every compilation unit is instrumented
(including AMReX/OpenPMD sub-builds pulled in via add_subdirectory).
* Drives the whole tests/ tree via `coverage run ... -m pytest` so
the new Wave C-prep tests (output validation, read round-trip,
write-var-normal-dist, driver unit, build-config-gated submodule
tests) all fire; optional-dep tests that did skip on my laptop will
run here because the submodules are built.
Coverage collection
* gcovr emits coverage-c.xml (Cobertura) + coverage-c.html (detail)
filtered to h5bench-owned directories (commons/, h5bench_patterns/,
exerciser/, metadata_stress/) so vendored submodule coverage
doesn't dominate the numbers.
* coverage.py emits coverage-python.xml scoped to src/ (the driver).
* codecov/codecov-action@v4 uploads both with -Fc / -Fpython flags
so the Codecov UI can present C vs Python breakdowns.
* CODECOV_TOKEN is read from secrets but the action tolerates it
being empty (tokenless upload still works for public repos).
* actions/upload-artifact@v4 archives the HTML + XML reports as a
CI artifact (14-day retention) for manual inspection when the
Codecov upload itself fails.
Local baseline (without submodule builds, recorded on wave-c-prep-tests):
C lines 55.7% -> 64.6%
C func 59.7% -> 72.5%
Python 37% -> 54%
With H5BENCH_ALL + WITH_ASYNC_VOL this workflow should push both past
80% because it exercises the run_amrex / run_openpmd / run_e3sm /
run_macsio dispatchers and the MODE=ASYNC branches.
The hpcio/hdf5-1.14.0 base image (Ubuntu 20.04) carries a Kitware apt source whose GPG key has rotated; running `apt-get update` inside container fails with: W: GPG error: https://apt.kitware.com/ubuntu focal InRelease: NO_PUBKEY E: The repository '...' is not signed. Error: Process completed with exit code 100. We were only running apt to install python3-pip + python3-venv. The image ships python3 + pip already, and Ubuntu 20.04 predates PEP 668's externally-managed-environment marker, so we don't need a venv at all — `pip install --user` works and writes to /root/.local. Putting $HOME/.local/bin on PATH makes the subsequent gcovr/coverage/pytest invocations resolve to the pinned versions. Mirrors the workaround already used in tasks/run_coverage_in_docker.sh when reproducing the workflow locally.
jeanbez
force-pushed
the
wave-c-prep-tests
branch
from
June 29, 2026 15:52
4c4a638 to
e8fb120
Compare
jeanbez
added a commit
that referenced
this pull request
Jul 30, 2026
The _full_setup() helper in tests/test_driver_unit.py (from #157) built its base config with empty-dict placeholders — {'mpi': {}, 'vol': {}, 'file-system': {}, 'directory': {}, 'benchmarks': []}. That satisfied the legacy top-level-keys check but the schema this PR introduces requires string-typed 'directory', mpi.command, and a non-empty benchmarks array with a valid oneOf entry. Rebuild _full_setup() using the opaque-benchmark oneOf branch (the least-restrictive way to satisfy the benchmarks entry) so: * test_validate_json_accepts_all_required_keys — passes because the minimum setup is now schema-valid. * test_validate_json_tolerates_extra_keys — still passes because the top-level schema has additionalProperties: true. * test_validate_json_exits_when_key_missing — still passes because removing any of the five required keys still trips the schema's 'required' rule and validate_json exits with EX_DATAERR.
jeanbez
added a commit
that referenced
this pull request
Aug 18, 2026
codecov/patch was blocking #158 because it defaults to demanding new lines meet or exceed the project average. h5bench's line-level coverage is still growing (Wave C-prep in #157 added Python coverage from ~0% up; the C binaries are mostly exercised by integration tests, not unit tests), so a percentage-based gate is premature. codecov.yml at repo root marks both project and patch status checks as informational. Codecov continues to annotate PRs with per-line coverage so the numbers are visible; the checks just do not block a merge. Revisit once coverage has a stable baseline.
jeanbez
added a commit
that referenced
this pull request
Aug 19, 2026
* Add JSON schema for configuration files + driver-side validation
The Python driver previously did `for p in [...]: if p not in setup` —
five top-level keys, nothing about benchmark dispatch, MODE / MEM_PATTERN /
FILE_PATTERN / READ_OPTION / COMPRESS / COLLECTIVE_* enums, numeric-string
discipline, or per-launcher MPI arg conventions. A config with `MODE: "BOGUS"`
or `MEM_PATTERN: "sequential"` ran clean through validate_json and only
surfaced as a binary-level "Unsupported" log at runtime.
This commit lands schemas/h5bench-config.schema.json — JSON Schema Draft
2020-12 — covering the entire configuration grammar:
* top-level: mpi / vol / file-system / directory / benchmarks required
* mpi.command: launcher enum (mpirun, mpiexec, srun, jsrun, runjob)
* benchmarks[].benchmark: enum of the 12 names the driver actually
dispatches on (catches the legacy "write_normal_dist" typo)
* pattern benchmarks (write / read / overwrite / append / write-unlimited
/ write_var_normal_dist): strictly typed configuration with MODE,
MEM_PATTERN, FILE_PATTERN, READ_OPTION, COLLECTIVE_DATA, COLLECTIVE_
METADATA, COMPRESS, ALIGN, NUM_DIMS enums; DIM_*, CHUNK_DIM_*, STRIDE_*,
BLOCK_*, ALIGN_*, TIMESTEPS, DELAYED_CLOSE_TIMESTEPS as numeric strings;
NUM_PARTICLES as `<n>` or `<n> Ks/Ms/Gs/Ts`; EMULATED_COMPUTE_TIME_PER_
TIMESTEP as `<n> <unit>` with unit ∈ {min, sec, s, ms, us}; CSV_FILE as
a string.
* exerciser / metadata / amrex / openpmd / e3sm / macsio: opaque
configuration objects (additionalProperties: true) since the binary
parses its own keys — we don't second-guess.
src/h5bench.py validate_json now:
1. soft-imports jsonschema (try/except ImportError), so existing
deployments without the package keep working
2. looks up the schema via $H5BENCH_SCHEMA env var, then relative to
__file__: `<dir>/schemas/`, `<dir>/../schemas/`,
`<dir>/../share/h5bench/schemas/` — covers source-tree, build-dir,
and install layouts without a CMake install rule yet
3. validates with jsonschema if available, logs the offending JSON path
("benchmarks/0/configuration/MODE") and exits EX_DATAERR on failure
4. falls back to the legacy 5-key check with a one-line "install
jsonschema for full validation" warning when the package is missing
Other changes:
* samples/sync-write-1d-contig-contig-normal-dist.json: long-standing
typo "write_normal_dist" -> "write_var_normal_dist". The driver only
recognises the var_ form, so the sample was silently falling through
to the "Unsupported benchmark/kernel" branch.
* tests/test_schema.py: 87 tests — every shipped sample validates,
each top-level key removed in turn fails, each pattern enum accepts
valid values and rejects garbage, the legacy `write_normal_dist`
name is rejected, dim values as actual numbers (rather than strings)
are rejected, opaque benchmarks pass through with arbitrary keys.
Driver-level tests confirm validate_json exits EX_DATAERR on schema
failure and that the jsonschema-missing fallback still rejects
missing top-level keys.
* tests/CMakeLists.txt: register h5bench-schema with ctest.
* tests/requirements.txt: pin jsonschema>=4.0.
* docs/source/running.rst: short paragraph pointing at the schema
file plus a one-liner Python command for out-of-band validation.
Verified locally: 87/87 schema tests pass in 1.22s; all 51 production
sample configs validate cleanly after the typo fix.
* Install jsonschema in the develop-test CI so test_schema actually runs
test_schema.py is registered with ctest (h5bench-schema), and the
h5bench-hdf5-develop-test.yml workflow is the only one that runs
h5bench's own ctest (cd build-sync && ctest, cd build-async && ctest).
That workflow installed only pytest, so test_schema.py's `import
jsonschema` would raise ModuleNotFoundError at collection time and fail
the h5bench-schema test red on every PR.
Two changes:
* h5bench-hdf5-develop-test.yml: add
`pip install -r tests/requirements.txt` next to the existing
`pip install pytest`, so jsonschema (and any future test dep) is
present when ctest runs the suite.
* test_schema.py: replace the bare `import jsonschema` with
`pytest.importorskip("jsonschema")`. Belt-and-suspenders — the
workflow change is the real fix (we want the test to RUN, not
skip), but importorskip means a missing dependency degrades to a
clean skip instead of a hard collection error.
The four production h5bench-hdf5-*.yml workflows do not run h5bench's
ctest (their `ctest` calls are inside cd $ASYNC_DIR/build — that's the
VOL-ASYNC project's own suite), so they are unaffected.
* Rebuild driver-unit fixture as a schema-valid config
The _full_setup() helper in tests/test_driver_unit.py (from #157) built
its base config with empty-dict placeholders — {'mpi': {}, 'vol': {},
'file-system': {}, 'directory': {}, 'benchmarks': []}. That satisfied
the legacy top-level-keys check but the schema this PR introduces
requires string-typed 'directory', mpi.command, and a non-empty
benchmarks array with a valid oneOf entry.
Rebuild _full_setup() using the opaque-benchmark oneOf branch (the
least-restrictive way to satisfy the benchmarks entry) so:
* test_validate_json_accepts_all_required_keys — passes because the
minimum setup is now schema-valid.
* test_validate_json_tolerates_extra_keys — still passes because the
top-level schema has additionalProperties: true.
* test_validate_json_exits_when_key_missing — still passes because
removing any of the five required keys still trips the schema's
'required' rule and validate_json exits with EX_DATAERR.
* Cover schema-loader fallback branches in driver-unit tests
Codecov flagged the new schema-loader paths in src/h5bench.py as
uncovered on this PR:
* _load_config_schema's ImportError branch (jsonschema missing)
* _load_config_schema's no-candidate-found branch
* _load_config_schema's H5BENCH_SCHEMA env-var override
* validate_json's legacy-fallback branch when the schema loader
returns (None, None)
* validate_json's fallback exit-on-missing-key path
Add five tests (one parametrised over the required keys) that
exercise each of those branches. All are pure Python, no HDF5, no
subprocess.
* Mark codecov status checks as informational
codecov/patch was blocking #158 because it defaults to demanding new
lines meet or exceed the project average. h5bench's line-level
coverage is still growing (Wave C-prep in #157 added Python coverage
from ~0% up; the C binaries are mostly exercised by integration
tests, not unit tests), so a percentage-based gate is premature.
codecov.yml at repo root marks both project and patch status checks
as informational. Codecov continues to annotate PRs with per-line
coverage so the numbers are visible; the checks just do not block a
merge. Revisit once coverage has a stable baseline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Structural refactor of the C benchmark sources. This PR puts the additional testing scaffolding in place. No functional change for users; entirely additive in
tests/plus an extension to the CMake-generatedh5bench_configuration.py.What's in the PR
Output content validation
tests/helpers/h5_assert.py— h5py-based assertion library:assert_timestep_count,assert_contig_datasets_present,assert_compound_dataset_present,assert_dataset_shape,assert_dataset_dtype,assert_unlimited_axis,assert_value_in_range,assert_id_1_1d_pattern,assert_id_2_1d_write_pattern,assert_datasets_equal,dataset_sha256.tests/fixtures/tiny-*.json— 11 small configs (1024 particles × 2 ranks × 1–2 timesteps) covering 1D write, the three non-CONTIG/CONTIGMEM × FILEpattern combinations, 2D write, 3D write, append, overwrite, unlimited, variable-distribution write, and write-then-read round trip.tests/test_output_validation.py— 9 tests verifying group/dataset presence, shape, dtype, value ranges, the deterministicid_1/id_2patterns,H5S_UNLIMITEDaxis survival across append, and the "overwrite actually changesid_2values" invariant.tests/test_read_roundtrip.py— closes theh5bench_readgap entirely (zero pytest coverage before).tests/test_write_var_normal_dist.py— replaces the legacytest_sync_h5bench_normal_dist.pywhich had two latent bugs (wrong binary name, glob for a non-existent sample) and had therefore never actually run.Python driver unit tests
tests/test_driver_unit.py— 30 pytest cases for the parts ofsrc/h5bench.pythat don't need a benchmark binary:validate_json(all 5 required keys + each missing-key path →EX_DATAERR, extras tolerated),prepare_parallel(mpirun/mpiexec/srun/explicit-config/unknown-command),is_available(hit/miss viashutil.which),check_parallel(plain shell /$SHELLismpirun/mpiexec/srun→EX_USAGE/$SHELLunset),prepare_vol/enable_vol/disable_vol/reset_vol(all branches),check_for_hdf5_error(clean stderr / banner →EX_IOERR),run()early-exit paths (missing config / malformed JSON →EX_NOINPUT). Runs in ~0.05 s total — no MPI, no HDF5, no subprocess.Build-option detection
src/h5bench_configuration.py.in— CMakeconfigure_filenow writes an__options__dict at configure time exposing everyH5BENCH_*flag +WITH_ASYNC_VOLas a real Python boolean.tests/helpers/build_config.py— onepytest.mark.skipifmarker per flag:requires_metadata,requires_exerciser,requires_amrex,requires_openpmd,requires_e3sm,requires_macsio,requires_async_vol. Reason strings include the flag state sopytest -vis self-explanatory:Submodule + ASYNC tests
test_sync_amrex.py,test_async_amrex.py,test_sync_openpmd.py,test_async_h5bench.py,test_sync_exerciser.py,test_sync_metadata.pyto use the new markers. Binary-missing is now a hard assert (build regression) rather than a silent skip, so a missingh5bench_amrex_syncwhenH5BENCH_AMREX=ONfails loudly.test_sync_e3sm.pyandtest_sync_macsio.py— these benchmarks had no pytest coverage at all before.CI coverage workflow
.github/workflows/coverage.yml— single dedicated job onhpcio/hdf5-1.14.0. Builds Argobots + VOL-ASYNC, then h5bench with-DH5BENCH_ALL=ON -DWITH_ASYNC_VOL:BOOL=ON --coverageon C/C++ flags + linker flags. Runs the wholetests/tree undercoverage run --branch -m pytest, collectsgcovr(C) +coverage.py(Python), uploads both to Codecov with-Fc/-Fpythonflags. Also archives HTML detail + XML toactions/upload-artifactwith 14-day retention. Triggers: PR todevelop/master, push tomaster, manual dispatch, weekly Monday drift check.Plumbing
tests/conftest.py— VOL-path injection extracted into a helper; now also processestests/fixtures/*.jsonso tiny configs get the same VOL-library treatment as production samples.tests/CMakeLists.txt— registersh5bench-output-validation,h5bench-read-roundtrip,h5bench-write-var-normal-dist,h5bench-driver-unitunconditionally;h5bench-sync-e3sm/h5bench-sync-macsiogated onH5BENCH_E3SM/H5BENCH_MACSIO.tests/requirements.txt—h5py>=3.0,numpy>=1.20.Coverage delta
Measured locally with a build that has
H5BENCH_EXERCISER,H5BENCH_METADATA, andH5BENCH_OPENPMDenabled (the Docker dry-run could not build AMReX, MACSio'sjson_cwxdep, or run 16M-particle production samples under x86 emulation):src/h5bench.pyA native x86 GH Actions runner with
H5BENCH_ALL=ONshould push both meaningfully higher — AMReX/MACSio/E3SM build there and the production samples finish in seconds rather than hours.Surprising invariants now locked in
The tests intentionally capture three currently-true behaviours so a Wave C refactor can't silently change them:
h5bench_writeclobbersX_DIM/Y_DIM/Z_DIMat runtime withX_RAND/Y_RAND/Z_RAND(191/1009/3701), so range checks use those values rather than the file-scope64defaults.appenddoes NOT grow the file beyond2 * dim_1. With 2 ranks each writingdim_1particles viawrite-unlimitedfirst, the file already has that extent, soappendoverwrites the second half.write-unlimitedrequiresCOMPRESS: "YES"(or explicit chunking) — unchunked datasets cannot haveH5S_UNLIMITEDaxes.Deferred to follow-ups
uniform_random_number(). Today's tests cover structure + deterministic-id patterns; full content checksums come later.configuration.json(Wave D item 22).Test plan
h5bench-hdf5-*.ymlworkflows aren't touched).Coverageworkflow turns green and uploads to Codecov. (May needCODECOV_TOKENsecret added in Settings → Secrets for reliable uploads under rate limiting; the action tolerates missing token for public-repo tokenless upload.)README.md(separate small PR).Local verification
5 new test files + 6 refactored tests run cleanly: 44 / 44 pass in the Docker dry-run with
H5BENCH_EXERCISER/METADATA/OPENPMD+WITH_ASYNC_VOLbuilt. Optional-dep tests skip with explicit per-flag reasons when the relevant CMake option is OFF.