Skip to content

Commit 94f11ed

Browse files
jeanbezjeanbez
authored
Add JSON schema for configuration files + driver-side validation (#158)
* Add JSON schema for configuration files + driver-side validation The Python driver previously did `for p in [...]: if p not in setup` — five top-level keys, nothing about benchmark dispatch, MODE / MEM_PATTERN / FILE_PATTERN / READ_OPTION / COMPRESS / COLLECTIVE_* enums, numeric-string discipline, or per-launcher MPI arg conventions. A config with `MODE: "BOGUS"` or `MEM_PATTERN: "sequential"` ran clean through validate_json and only surfaced as a binary-level "Unsupported" log at runtime. This commit lands schemas/h5bench-config.schema.json — JSON Schema Draft 2020-12 — covering the entire configuration grammar: * top-level: mpi / vol / file-system / directory / benchmarks required * mpi.command: launcher enum (mpirun, mpiexec, srun, jsrun, runjob) * benchmarks[].benchmark: enum of the 12 names the driver actually dispatches on (catches the legacy "write_normal_dist" typo) * pattern benchmarks (write / read / overwrite / append / write-unlimited / write_var_normal_dist): strictly typed configuration with MODE, MEM_PATTERN, FILE_PATTERN, READ_OPTION, COLLECTIVE_DATA, COLLECTIVE_ METADATA, COMPRESS, ALIGN, NUM_DIMS enums; DIM_*, CHUNK_DIM_*, STRIDE_*, BLOCK_*, ALIGN_*, TIMESTEPS, DELAYED_CLOSE_TIMESTEPS as numeric strings; NUM_PARTICLES as `<n>` or `<n> Ks/Ms/Gs/Ts`; EMULATED_COMPUTE_TIME_PER_ TIMESTEP as `<n> <unit>` with unit ∈ {min, sec, s, ms, us}; CSV_FILE as a string. * exerciser / metadata / amrex / openpmd / e3sm / macsio: opaque configuration objects (additionalProperties: true) since the binary parses its own keys — we don't second-guess. src/h5bench.py validate_json now: 1. soft-imports jsonschema (try/except ImportError), so existing deployments without the package keep working 2. looks up the schema via $H5BENCH_SCHEMA env var, then relative to __file__: `<dir>/schemas/`, `<dir>/../schemas/`, `<dir>/../share/h5bench/schemas/` — covers source-tree, build-dir, and install layouts without a CMake install rule yet 3. validates with jsonschema if available, logs the offending JSON path ("benchmarks/0/configuration/MODE") and exits EX_DATAERR on failure 4. falls back to the legacy 5-key check with a one-line "install jsonschema for full validation" warning when the package is missing Other changes: * samples/sync-write-1d-contig-contig-normal-dist.json: long-standing typo "write_normal_dist" -> "write_var_normal_dist". The driver only recognises the var_ form, so the sample was silently falling through to the "Unsupported benchmark/kernel" branch. * tests/test_schema.py: 87 tests — every shipped sample validates, each top-level key removed in turn fails, each pattern enum accepts valid values and rejects garbage, the legacy `write_normal_dist` name is rejected, dim values as actual numbers (rather than strings) are rejected, opaque benchmarks pass through with arbitrary keys. Driver-level tests confirm validate_json exits EX_DATAERR on schema failure and that the jsonschema-missing fallback still rejects missing top-level keys. * tests/CMakeLists.txt: register h5bench-schema with ctest. * tests/requirements.txt: pin jsonschema>=4.0. * docs/source/running.rst: short paragraph pointing at the schema file plus a one-liner Python command for out-of-band validation. Verified locally: 87/87 schema tests pass in 1.22s; all 51 production sample configs validate cleanly after the typo fix. * Install jsonschema in the develop-test CI so test_schema actually runs test_schema.py is registered with ctest (h5bench-schema), and the h5bench-hdf5-develop-test.yml workflow is the only one that runs h5bench's own ctest (cd build-sync && ctest, cd build-async && ctest). That workflow installed only pytest, so test_schema.py's `import jsonschema` would raise ModuleNotFoundError at collection time and fail the h5bench-schema test red on every PR. Two changes: * h5bench-hdf5-develop-test.yml: add `pip install -r tests/requirements.txt` next to the existing `pip install pytest`, so jsonschema (and any future test dep) is present when ctest runs the suite. * test_schema.py: replace the bare `import jsonschema` with `pytest.importorskip("jsonschema")`. Belt-and-suspenders — the workflow change is the real fix (we want the test to RUN, not skip), but importorskip means a missing dependency degrades to a clean skip instead of a hard collection error. The four production h5bench-hdf5-*.yml workflows do not run h5bench's ctest (their `ctest` calls are inside cd $ASYNC_DIR/build — that's the VOL-ASYNC project's own suite), so they are unaffected. * Rebuild driver-unit fixture as a schema-valid config The _full_setup() helper in tests/test_driver_unit.py (from #157) built its base config with empty-dict placeholders — {'mpi': {}, 'vol': {}, 'file-system': {}, 'directory': {}, 'benchmarks': []}. That satisfied the legacy top-level-keys check but the schema this PR introduces requires string-typed 'directory', mpi.command, and a non-empty benchmarks array with a valid oneOf entry. Rebuild _full_setup() using the opaque-benchmark oneOf branch (the least-restrictive way to satisfy the benchmarks entry) so: * test_validate_json_accepts_all_required_keys — passes because the minimum setup is now schema-valid. * test_validate_json_tolerates_extra_keys — still passes because the top-level schema has additionalProperties: true. * test_validate_json_exits_when_key_missing — still passes because removing any of the five required keys still trips the schema's 'required' rule and validate_json exits with EX_DATAERR. * Cover schema-loader fallback branches in driver-unit tests Codecov flagged the new schema-loader paths in src/h5bench.py as uncovered on this PR: * _load_config_schema's ImportError branch (jsonschema missing) * _load_config_schema's no-candidate-found branch * _load_config_schema's H5BENCH_SCHEMA env-var override * validate_json's legacy-fallback branch when the schema loader returns (None, None) * validate_json's fallback exit-on-missing-key path Add five tests (one parametrised over the required keys) that exercise each of those branches. All are pure Python, no HDF5, no subprocess. * Mark codecov status checks as informational codecov/patch was blocking #158 because it defaults to demanding new lines meet or exceed the project average. h5bench's line-level coverage is still growing (Wave C-prep in #157 added Python coverage from ~0% up; the C binaries are mostly exercised by integration tests, not unit tests), so a percentage-based gate is premature. codecov.yml at repo root marks both project and patch status checks as informational. Codecov continues to annotate PRs with per-line coverage so the numbers are visible; the checks just do not block a merge. Revisit once coverage has a stable baseline.
1 parent 59d2765 commit 94f11ed

10 files changed

Lines changed: 726 additions & 13 deletions

File tree

‎.github/workflows/h5bench-hdf5-develop-test.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ jobs:
3636
git clone --recursive https://github.com/hpc-io/vol-async.git --branch develop /opt/vol-async
3737
3838
python3 -m pip install pytest
39+
python3 -m pip install -r tests/requirements.txt
3940
4041
- name: Build HDF5 develop
4142
run: |

‎codecov.yml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# codecov configuration
2+
#
3+
# h5bench's test suite is expanding gradually (Wave C-prep in #157
4+
# added Python-side coverage from ~0% up; the C benchmark binaries
5+
# themselves are mostly exercised by integration tests, not unit
6+
# tests, so line-level coverage is inherently modest). The default
7+
# "auto" target for codecov/patch demands new lines meet or exceed
8+
# the project average, which is too strict while coverage is still
9+
# growing.
10+
#
11+
# Mark both status checks as informational: codecov continues to
12+
# annotate PRs with per-line coverage, but does not block merges on
13+
# a coverage-percentage target. Revisit once project coverage has
14+
# a stable baseline.
15+
16+
coverage:
17+
status:
18+
project:
19+
default:
20+
informational: true
21+
patch:
22+
default:
23+
informational: true

‎docs/source/running.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,12 @@ You can find several samples of configuration file with all the options in our [
241241

242242
For a description of all the options available in each benchmark, please refer to their entries in the documentation.
243243

244+
The full configuration grammar is captured in a JSON Schema file at ``schemas/h5bench-config.schema.json`` (root of the repository). When the optional ``jsonschema`` Python package is installed (``pip install jsonschema``), the driver validates every config against this schema on startup and rejects unknown keys, wrong-typed values, and out-of-enum settings (for example ``MODE: "BOGUS"`` or ``MEM_PATTERN: "sequential"``) with a precise error message pointing at the offending field. Without ``jsonschema``, the driver falls back to a minimal five-required-keys check and prints a warning.
245+
246+
You can run the schema against a config out-of-band before launching::
247+
248+
python3 -c "import json, jsonschema; jsonschema.validate(json.load(open('my.json')), json.load(open('schemas/h5bench-config.schema.json')))"
249+
244250
When the ``--debug`` option is enabled, you can expect an output similar to:
245251

246252
.. code-block::

‎samples/sync-write-1d-contig-contig-normal-dist.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
"directory": "storage",
1212
"benchmarks": [
1313
{
14-
"benchmark": "write_normal_dist",
14+
"benchmark": "write_var_normal_dist",
1515
"file": "test.h5",
1616
"configuration": {
1717
"MEM_PATTERN": "CONTIG",

‎schemas/h5bench-config.schema.json‎

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
{
2+
"$schema": "https://json-schema.org/draft/2020-12/schema",
3+
"$id": "https://github.com/hpc-io/h5bench/schemas/h5bench-config.schema.json",
4+
"title": "h5bench configuration",
5+
"description": "Validates the JSON file passed to the h5bench Python driver. Pattern benchmarks (write/read/append/overwrite/write-unlimited/write_var_normal_dist) are strictly typed because the driver translates their keys into argv for the binary; exerciser/metadata/amrex/openpmd/e3sm/macsio carry an opaque configuration object because their binaries parse the keys themselves.",
6+
"type": "object",
7+
"required": ["mpi", "vol", "file-system", "directory", "benchmarks"],
8+
"additionalProperties": true,
9+
"properties": {
10+
"mpi": { "$ref": "#/$defs/mpiSection" },
11+
"vol": { "$ref": "#/$defs/volSection" },
12+
"file-system": { "$ref": "#/$defs/fileSystemSection" },
13+
"directory": {
14+
"type": "string",
15+
"minLength": 1,
16+
"description": "Output directory for benchmark artifacts (created if missing)."
17+
},
18+
"benchmarks": {
19+
"type": "array",
20+
"minItems": 1,
21+
"items": { "$ref": "#/$defs/benchmarkEntry" }
22+
}
23+
},
24+
"$defs": {
25+
"mpiSection": {
26+
"type": "object",
27+
"required": ["command"],
28+
"additionalProperties": false,
29+
"properties": {
30+
"command": {
31+
"type": "string",
32+
"enum": ["mpirun", "mpiexec", "srun", "jsrun", "runjob"],
33+
"description": "MPI launcher binary. The driver wires up -np / -n for mpirun/mpiexec/srun automatically when 'configuration' is absent."
34+
},
35+
"ranks": {
36+
"type": "string",
37+
"pattern": "^\\d+$",
38+
"description": "Rank count as a numeric string. Used only when 'configuration' is absent."
39+
},
40+
"configuration": {
41+
"type": "string",
42+
"description": "Raw extra arguments appended verbatim after the launcher command. When set, overrides the implicit -np <ranks> wiring."
43+
}
44+
}
45+
},
46+
"volSection": {
47+
"type": "object",
48+
"additionalProperties": false,
49+
"properties": {
50+
"library": {
51+
"type": "string",
52+
"description": "Colon-separated list of directories prepended to LD_LIBRARY_PATH (and DYLD_LIBRARY_PATH) for the run."
53+
},
54+
"preload": {
55+
"type": "string",
56+
"description": "Colon-separated list of shared objects prepended to LD_PRELOAD."
57+
},
58+
"path": {
59+
"type": "string",
60+
"description": "Directory exported as HDF5_PLUGIN_PATH so HDF5 finds VOL connectors."
61+
},
62+
"connector": {
63+
"type": "string",
64+
"description": "Value for HDF5_VOL_CONNECTOR, e.g. 'async under_vol=0;under_info={}'."
65+
}
66+
}
67+
},
68+
"fileSystemSection": {
69+
"type": "object",
70+
"additionalProperties": false,
71+
"properties": {
72+
"lustre": {
73+
"type": "object",
74+
"additionalProperties": false,
75+
"properties": {
76+
"stripe-size": { "type": ["string", "integer"] },
77+
"stripe-count": { "type": ["string", "integer"] }
78+
}
79+
}
80+
}
81+
},
82+
83+
"benchmarkEntry": {
84+
"type": "object",
85+
"required": ["benchmark"],
86+
"properties": {
87+
"benchmark": {
88+
"type": "string",
89+
"enum": [
90+
"write",
91+
"write-unlimited",
92+
"overwrite",
93+
"append",
94+
"read",
95+
"write_var_normal_dist",
96+
"exerciser",
97+
"metadata",
98+
"amrex",
99+
"openpmd",
100+
"e3sm",
101+
"macsio"
102+
]
103+
},
104+
"file": {
105+
"type": "string",
106+
"description": "Output filename relative to the top-level 'directory'."
107+
},
108+
"configuration": { "type": "object" }
109+
},
110+
"oneOf": [
111+
{ "$ref": "#/$defs/patternBenchmark" },
112+
{ "$ref": "#/$defs/exerciserBenchmark" },
113+
{ "$ref": "#/$defs/metadataBenchmark" },
114+
{ "$ref": "#/$defs/opaqueBenchmark" }
115+
]
116+
},
117+
118+
"patternBenchmark": {
119+
"description": "write / write-unlimited / overwrite / append / read / write_var_normal_dist — the driver translates the configuration keys into argv for the h5bench_<pattern> binary, so each key is strictly typed.",
120+
"type": "object",
121+
"properties": {
122+
"benchmark": {
123+
"enum": [
124+
"write",
125+
"write-unlimited",
126+
"overwrite",
127+
"append",
128+
"read",
129+
"write_var_normal_dist"
130+
]
131+
},
132+
"file": { "type": "string" },
133+
"configuration": { "$ref": "#/$defs/patternConfiguration" }
134+
},
135+
"required": ["benchmark", "file", "configuration"],
136+
"additionalProperties": false
137+
},
138+
139+
"patternConfiguration": {
140+
"type": "object",
141+
"additionalProperties": false,
142+
"properties": {
143+
"MEM_PATTERN": { "enum": ["CONTIG", "INTERLEAVED", "STRIDED"] },
144+
"FILE_PATTERN": { "enum": ["CONTIG", "INTERLEAVED", "STRIDED"] },
145+
"MODE": { "enum": ["SYNC", "ASYNC", "LOG"] },
146+
"READ_OPTION": { "enum": ["FULL", "PARTIAL", "STRIDED", "LDC", "RDC", "CS", "PRL"] },
147+
"COLLECTIVE_DATA": { "enum": ["YES", "NO"] },
148+
"COLLECTIVE_METADATA": { "enum": ["YES", "NO"] },
149+
"COMPRESS": { "enum": ["YES", "NO"] },
150+
"ALIGN": { "enum": ["YES", "NO"] },
151+
152+
"NUM_DIMS": { "$ref": "#/$defs/dimCount" },
153+
"DIM_1": { "$ref": "#/$defs/numericString" },
154+
"DIM_2": { "$ref": "#/$defs/numericString" },
155+
"DIM_3": { "$ref": "#/$defs/numericString" },
156+
"CHUNK_DIM_1": { "$ref": "#/$defs/numericString" },
157+
"CHUNK_DIM_2": { "$ref": "#/$defs/numericString" },
158+
"CHUNK_DIM_3": { "$ref": "#/$defs/numericString" },
159+
"STDEV_DIM_1": { "$ref": "#/$defs/numericString" },
160+
161+
"TIMESTEPS": { "$ref": "#/$defs/numericString" },
162+
"DELAYED_CLOSE_TIMESTEPS": { "$ref": "#/$defs/numericString" },
163+
164+
"STRIDE_SIZE": { "$ref": "#/$defs/numericString" },
165+
"STRIDE_SIZE_2": { "$ref": "#/$defs/numericString" },
166+
"STRIDE_SIZE_3": { "$ref": "#/$defs/numericString" },
167+
"BLOCK_SIZE": { "$ref": "#/$defs/numericString" },
168+
"BLOCK_SIZE_2": { "$ref": "#/$defs/numericString" },
169+
"BLOCK_SIZE_3": { "$ref": "#/$defs/numericString" },
170+
"BLOCK_CNT": { "$ref": "#/$defs/numericString" },
171+
172+
"ALIGN_LEN": { "$ref": "#/$defs/numericString" },
173+
"ALIGN_THRESHOLD": { "$ref": "#/$defs/numericString" },
174+
175+
"NUM_PARTICLES": {
176+
"type": "string",
177+
"pattern": "^\\d+(\\s+(K|M|G|T)s?)?$",
178+
"description": "Accepts either a plain integer ('1024') or the K/M/G/T suffix form h5bench parses ('1024 Ks')."
179+
},
180+
181+
"EMULATED_COMPUTE_TIME_PER_TIMESTEP": {
182+
"type": "string",
183+
"pattern": "^\\d+\\s+(min|sec|s|ms|us)$",
184+
"description": "<value> <unit>. Unit ∈ {min, sec, s, ms, us}; '0 s' is valid for fast tests."
185+
},
186+
187+
"CSV_FILE": { "type": "string" }
188+
}
189+
},
190+
191+
"dimCount": {
192+
"type": "string",
193+
"enum": ["1", "2", "3"],
194+
"description": "Number of dataset dimensions; 1, 2, or 3."
195+
},
196+
197+
"numericString": {
198+
"type": "string",
199+
"pattern": "^\\d+$",
200+
"description": "h5bench config historically uses string-quoted integers, e.g. \"1024\"."
201+
},
202+
203+
"exerciserBenchmark": {
204+
"description": "h5bench_exerciser receives its configuration as argv flags; the driver passes them through verbatim.",
205+
"type": "object",
206+
"properties": {
207+
"benchmark": { "const": "exerciser" },
208+
"configuration": { "type": "object" }
209+
},
210+
"required": ["benchmark"],
211+
"additionalProperties": false
212+
},
213+
214+
"metadataBenchmark": {
215+
"description": "h5bench_hdf5_iotest writes its own ini-style config; the driver passes a 'file' name and the configuration block.",
216+
"type": "object",
217+
"properties": {
218+
"benchmark": { "const": "metadata" },
219+
"file": { "type": "string" },
220+
"configuration": { "type": "object" }
221+
},
222+
"required": ["benchmark", "configuration"],
223+
"additionalProperties": false
224+
},
225+
226+
"opaqueBenchmark": {
227+
"description": "amrex / openpmd / e3sm / macsio — externally-owned benchmark binaries that parse their own configuration; the schema does not introspect their keys.",
228+
"type": "object",
229+
"properties": {
230+
"benchmark": { "enum": ["amrex", "openpmd", "e3sm", "macsio"] },
231+
"file": { "type": "string" },
232+
"configuration": { "type": "object" }
233+
},
234+
"required": ["benchmark", "configuration"],
235+
"additionalProperties": false
236+
}
237+
}
238+
}

‎src/h5bench.py‎

Lines changed: 62 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -158,21 +158,72 @@ def prepare(self, setup):
158158
self.logger.info('Lustre support not detected')
159159

160160
def validate_json(self, setup):
161-
"""Make sure JSON contains all the necessary properties."""
162-
properties = [
163-
'mpi',
164-
'vol',
165-
'file-system',
166-
'directory',
167-
'benchmarks'
168-
]
161+
"""Validate the JSON configuration against the bundled JSON schema.
162+
163+
Uses ``schemas/h5bench-config.schema.json`` (located next to or one
164+
level above this script, with an ``H5BENCH_SCHEMA`` env-var override)
165+
and the optional ``jsonschema`` package for full type/enum/regex
166+
validation. When ``jsonschema`` or the schema file is unavailable,
167+
falls back to the legacy five-required-keys check so existing setups
168+
keep working without a new pip dependency.
169+
"""
170+
schema, jsonschema_mod = self._load_config_schema()
171+
172+
if schema is not None and jsonschema_mod is not None:
173+
try:
174+
jsonschema_mod.validate(setup, schema)
175+
return
176+
except jsonschema_mod.ValidationError as e:
177+
location = '/'.join(str(p) for p in e.absolute_path) or '<root>'
178+
self.logger.critical(
179+
'JSON configuration invalid at "%s": %s', location, e.message
180+
)
181+
sys.exit(os.EX_DATAERR)
169182

170-
for p in properties:
171-
if p not in setup:
172-
self.logger.critical('JSON configuration file is invalid: "{}" property is missing'.format(p))
183+
if schema is None or jsonschema_mod is None:
184+
self.logger.warning(
185+
'jsonschema or schema file unavailable; falling back to a '
186+
'minimal top-level key check. Install jsonschema '
187+
'(pip install jsonschema) for full validation.'
188+
)
173189

190+
for p in ('mpi', 'vol', 'file-system', 'directory', 'benchmarks'):
191+
if p not in setup:
192+
self.logger.critical(
193+
'JSON configuration file is invalid: "{}" property is missing'.format(p)
194+
)
174195
sys.exit(os.EX_DATAERR)
175196

197+
def _load_config_schema(self):
198+
"""Locate ``h5bench-config.schema.json`` and the ``jsonschema`` module.
199+
200+
Returns (schema_dict, jsonschema_module). Either element may be None
201+
when the file or the package is missing — callers fall back to the
202+
legacy validation in that case.
203+
"""
204+
try:
205+
import jsonschema as jsonschema_mod
206+
except ImportError:
207+
return None, None
208+
209+
here = os.path.dirname(os.path.abspath(__file__))
210+
candidates = []
211+
env_path = os.environ.get('H5BENCH_SCHEMA')
212+
if env_path:
213+
candidates.append(env_path)
214+
candidates.extend([
215+
os.path.join(here, 'schemas', 'h5bench-config.schema.json'),
216+
os.path.join(here, '..', 'schemas', 'h5bench-config.schema.json'),
217+
os.path.join(here, '..', 'share', 'h5bench', 'schemas', 'h5bench-config.schema.json'),
218+
])
219+
220+
for path in candidates:
221+
if path and os.path.isfile(path):
222+
with open(path) as f:
223+
return json.load(f), jsonschema_mod
224+
225+
return None, jsonschema_mod
226+
176227
def run(self):
177228
"""Run all the benchmarks/kernels."""
178229
self.logger.info('Starting h5bench Suite')

‎tests/CMakeLists.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,14 @@ if(Python3_Interpreter_FOUND)
5252
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
5353
)
5454

55+
# JSON schema validation tests — pure Python, no benchmark binary needed.
56+
# Requires the `jsonschema` pip package (see tests/requirements.txt).
57+
add_test(
58+
NAME "h5bench-schema"
59+
COMMAND Python3::Interpreter -m pytest --verbose --rootdir ${CMAKE_BINARY_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/test_schema.py
60+
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
61+
)
62+
5563
if(H5BENCH_EXERCISER)
5664
add_test(
5765
NAME "h5bench-sync-exerciser"

‎tests/requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
h5py>=3.0
22
numpy>=1.20
3+
jsonschema>=4.0

0 commit comments

Comments
 (0)