Dataset-specific JSON/TOML configuration #1646

franzpoeschel · 2024-07-10T09:42:53Z

This was initially prototyped in PIConGPU, search for "extended format for JSON options" on this documentation page.

This moves that functionality over into openPMD.
Look at the diff on examples/13_write_dynamic_configuration.cpp for an example use case.
Bringing this to openPMD makes this functionality available in codes other than PIConGPU, and makes it available from TOML as well as from JSON.

TODO:

Document
Test
The NVIDIA compiler cant reliably parse TOML, don't let it run example 13
Check Python bindings for MyPath::openPMDPath()
Check JSON matching for OPEN_DATASET in ADIOS2
Merge Random-access for variable-based encoding (i.e. use SetStepSelection for ADIOS2 steps) #1706 first

src/IO/AbstractIOHandler.cpp

src/auxiliary/JSONMatcher.cpp

+and is interpreted as a regular expression against which the dataset name
+(full path or path within an iteration) must match.)END";
+        auto throw_up = [&](std::string const &additional_info,
+                            auto &&...additional_path) {


src/auxiliary/JSONMatcher.cpp

+and is interpreted as a regular expression against which the dataset name
+(full path or path within an iteration) must match.)END";
+        auto throw_up = [&](std::string const &additional_info,
+                            auto &&...additional_path) {


src/auxiliary/JSONMatcher.cpp

+and is interpreted as a regular expression against which the dataset name
+(full path or path within an iteration) must match.)END";
+        auto throw_up = [&](std::string const &additional_info,
+                            auto &&...additional_path) {


src/auxiliary/JSONMatcher.cpp

+
+namespace
+{
+    auto readPattern(


src/auxiliary/JSON.cpp

+std::string merge_impl(
+    std::string const &defaultValue,
+    std::string const &overwrite,
+    MPI_Comm_t &&...comm)


src/auxiliary/JSON.cpp

+std::string merge_impl(
+    std::string const &defaultValue,
+    std::string const &overwrite,
+    MPI_Comm_t &&...comm)


docs/source/details/backendconfig.rst

ax3l · 2025-03-20T23:53:09Z

examples/13_write_dynamic_configuration.cpp


    using position_t = double;
+
+#if !__NVCOMPILER // see https://github.com/ToruNiina/toml11/issues/205


Do you want to filter by NVHPC version, since the issue is fixed in newer releases?
ToruNiina/toml11#205 (comment)

Also, I think you meant to write

Suggested change

#if !__NVCOMPILER // see https://github.com/ToruNiina/toml11/issues/205

#if !defined(__NVCOMPILER) // see https://github.com/ToruNiina/toml11/issues/205

I'll replace it with the macro used in src/config.cpp:

#if !defined(__NVCOMPILER_MAJOR__) || __NVCOMPILER_MAJOR__ >= 23

ax3l

LGTM - thank you! 🎉

Please review the changes to docs I made.
Please see last inline comment on NVHPC.

might also be good for documentation purposes as JSON is more widely known

Having the same name as the public function provoked errors due to conversion from nlohmann::json types.

franzpoeschel added the api: new additions to the API label Jul 10, 2024

franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from 3d5b70a to 37d57ab Compare July 10, 2024 11:23

github-advanced-security bot found potential problems Jul 10, 2024

View reviewed changes

franzpoeschel force-pushed the dataset-specific-config branch 4 times, most recently from c95a953 to 68dc005 Compare July 16, 2024 11:12

github-advanced-security bot found potential problems Jul 16, 2024

View reviewed changes

ax3l modified the milestones: 0.16.0, 0.17.0 Jul 16, 2024

ax3l self-requested a review July 16, 2024 17:23

franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from c0f942e to fb7a28e Compare July 19, 2024 09:39

franzpoeschel force-pushed the dataset-specific-config branch from fb7a28e to 5f77a13 Compare August 5, 2024 09:30

franzpoeschel force-pushed the dataset-specific-config branch from 5f77a13 to 64b006d Compare October 1, 2024 13:38

franzpoeschel force-pushed the dataset-specific-config branch from 64b006d to 8852258 Compare October 29, 2024 08:19

franzpoeschel force-pushed the dataset-specific-config branch from 8852258 to a162e8d Compare November 15, 2024 15:01

franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from 7dda24d to 83e436c Compare December 17, 2024 11:19

franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from bdc319f to 0650789 Compare February 12, 2025 13:17

franzpoeschel mentioned this pull request Feb 13, 2025

Random-access for variable-based encoding (i.e. use SetStepSelection for ADIOS2 steps) #1706

Merged

7 tasks

franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from 6b2b5ee to 71536e6 Compare February 18, 2025 18:09

ax3l self-assigned this Feb 18, 2025

franzpoeschel force-pushed the dataset-specific-config branch from 71536e6 to 5e87f47 Compare February 20, 2025 18:16

ax3l reviewed Mar 20, 2025

View reviewed changes

docs/source/details/backendconfig.rst Outdated Show resolved Hide resolved

docs/source/details/backendconfig.rst Outdated Show resolved Hide resolved

ax3l reviewed Mar 20, 2025

View reviewed changes

ax3l approved these changes Mar 21, 2025

View reviewed changes

franzpoeschel and others added 26 commits March 25, 2025 18:12

Add JSONMatcher class

538c99a

Embed JSONMatcher into the backends

eabefdc

First attempt: Dataset-specific configuration

59545a9

Seems to work

c2de046

Adapt Coretest to new output of myPath()

74715fd

Better error messages and some documentation inside example

d496b0e

Adapt constructors for installation without ADIOS2/HDF5

66ae9a8

CI fixes

0b65178

Basic implementation

92c6ced

Update documentation and tests

61d4de6

Add a JSON translation of the config for NVHPC compilers

6d74a53

might also be good for documentation purposes as JSON is more widely known

Use dataset-specific config in tests

90d3a64

Fix: do_prune parameter for merge()

d7ceaba

Rename merge() -> merge_internal()

9121e0b

Having the same name as the public function provoked errors due to conversion from nlohmann::json types.

Don't compute the matchers for all backends

33f8e30

Add default block to test configs

067b1a6

Documentation

2dce165

Add TOML example

59206bb

Add Python binding for openPMD_path

56336f3

Fix doxygen

46f480a

Read dataset-specific configuration also in ADIOS2::openDataset

3877f2f

Cleanup

63a3dad

Fix initialization from Dummy IO Handler

458b4c5

Fix Doxygen

9c7efb4

Documentation Update

75df987

Fix NVCOMPILER macro in example

5e1e176

franzpoeschel force-pushed the dataset-specific-config branch from 3167802 to 5e1e176 Compare March 25, 2025 17:18

franzpoeschel enabled auto-merge (squash) March 25, 2025 17:18

franzpoeschel merged commit 12e1bb6 into openPMD:dev Mar 25, 2025
30 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Dataset-specific JSON/TOML configuration #1646

Dataset-specific JSON/TOML configuration #1646

Uh oh!

franzpoeschel commented Jul 10, 2024 •

edited

Loading

Uh oh!

Uh oh!

Check notice

Check notice

Check notice

Check warning

Check notice

Check notice

Uh oh!

Uh oh!

ax3l Mar 20, 2025 •

edited

Loading

Uh oh!

franzpoeschel Mar 25, 2025

Uh oh!

ax3l left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		using position_t = double;

		#if !__NVCOMPILER // see https://github.com/ToruNiina/toml11/issues/205

	#if !__NVCOMPILER // see https://github.com/ToruNiina/toml11/issues/205
	#if !defined(__NVCOMPILER) // see https://github.com/ToruNiina/toml11/issues/205

Dataset-specific JSON/TOML configuration #1646

Dataset-specific JSON/TOML configuration #1646

Uh oh!

Conversation

franzpoeschel commented Jul 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Check notice

Check notice

Check notice

Uh oh!

Check warning

Check notice

Check notice

Uh oh!

Uh oh!

ax3l Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

franzpoeschel Mar 25, 2025

Choose a reason for hiding this comment

Uh oh!

ax3l left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

franzpoeschel commented Jul 10, 2024 •

edited

Loading

ax3l Mar 20, 2025 •

edited

Loading