Random-access for variable-based encoding (i.e. use SetStepSelection for ADIOS2 steps) (#1706)

* Add preparseSnapshots

* Serial ADIOS2 implementation for readAttributeAllsteps

* Feature-complete, untested

* Fixes

* wip: parallel impl. for readAttributeAllsteps

huh this works??

* Simplify, cleanup

* wip testing

* CI fixes

* Fix test

* Add MPI_CHECK

* Stricter warning about group-based encoding in BP5

* variableBased as default in ADIOS2

* Add error check for non-homogeneous datasets

* Reset steps before reading the rankTable

* Use writeIterations() in MPI Benchmark

* Use group tables by default

* Add iterate_nonstreaming_series test to parallel tests

This covers the use case that the snapshot attribute has more than just
one entry.

* Remove std::cout calls

* wip: use variable-based encoding in tests

* CI fixes

* Fix Windows testing

* Cleanup

* Warn unimplemented modifiable attributes for now

* Fix test???

* Add this to variableBasedSeries test

* Remove debugging line

* Use own flag for the warning

Previously implementation did not work: ornladios/ADIOS2#4466

* Documentation

* Skip inhomogeneous datasets instead of outright failing

* add test for default iteration encoding

* Move test to its own file

* Don't distinguish ADIOS2 v2.9 any more

* Some more documentation

Author: franzpoeschel

CMakeLists.txt

@@ -787,6 +787,15 @@ if(openPMD_BUILD_TESTING)
                 test/Files_SerialIO/close_and_reopen_test.cpp
                 test/Files_SerialIO/filebased_write_test.cpp
             )
+        elseif(${test_name} STREQUAL "ParallelIO" AND openPMD_HAVE_MPI)
+            list(APPEND ${out_list}
+                test/Files_ParallelIO/read_variablebased_randomaccess.cpp
+                test/Files_ParallelIO/iterate_nonstreaming_series.cpp
+            )
+        elseif(${test_name} STREQUAL "Core")
+            list(APPEND ${out_list}
+                test/Files_Core/automatic_variable_encoding.cpp
+            )
         endif()
     endmacro()
 

docs/source/usage/concepts.rst

@@ -48,7 +48,7 @@ openPMD-api implements various file-formats (backends) and encoding strategies f
 **Iteration encoding:** The openPMD-api can encode iterations in different ways.
 The method ``Series::setIterationEncoding()`` (C++) or ``Series.set_iteration_encoding()`` (Python) may be used in writing for selecting one of the following encodings explicitly:
 
-* **group-based iteration encoding:** This encoding is the default.
+* **group-based iteration encoding:** This encoding is the default for HDF5 and JSON. In ADIOS2, variable-based encoding is preferred when possible due to better performance characteristics, see below.
   It creates a separate group in the hierarchy of the openPMD standard for each iteration.
   As an example, all data pertaining to iteration 0 may be found in group ``/data/0``, for iteration 100 in ``/data/100``.
 * **file-based iteration encoding:** A unique file on the filesystem is created for each iteration.
@@ -57,6 +57,7 @@ The method ``Series::setIterationEncoding()`` (C++) or ``Series.set_iteration_en
   A padding may be specified by ``"series_%06T.json"`` to create files ``series_000000.json``, ``series_000100.json`` and ``series_000200.json``.
   The inner group layout of each file is identical to that of the group-based encoding.
 * **variable-based iteration encoding:** This experimental encoding uses a feature of some backends (i.e., ADIOS2) to maintain datasets and attributes in several versions (i.e., iterations are stored inside *variables*).
+  When creating an ADIOS2 Series with steps (e.g. via ``series.writeIterations()`` / ``series.write_iterations()``), this encoding will be picked as a default instead of group-based encoding due to bad performance characteristics of group-based encoding in ADIOS2.
   No iteration-specific groups are created and the corresponding layer is dropped from the openPMD hierarchy.
   In backends that do not support this feature, a series created with this encoding can only contain one iteration.
 

docs/source/usage/workflow.rst

@@ -76,10 +76,14 @@ The openPMD-api distinguishes between a number of different access modes:
   3. In streaming backends, random-access is not possible.
      When using such a backend, the access mode will be coerced automatically to *linear read mode*.
      Use of Series::readIterations() is mandatory for access.
-  4. Reading a variable-based Series is only fully supported with *linear access mode*.
-     If using *random-access read mode*, the dataset will be considered to only have one single step.
-     If the dataset only has one single step, this is guaranteed to work as expected.
-     Otherwise, it is undefined which step's data is returned.
+  4. *Random-access read mode* for a variable-based Series is currently experimental.
+     There is currently only very restricted support for metadata definitions that change across steps:
+
+     1. Modifiable attributes (except ``/data/snapshot``) can currently not be read. Attributes such as ``/data/time`` that naturally change their value across Iterations will hence not be really well-usable; the last Iteration's value will currently leak into all other Iterations.
+     2. There is no support for datasets that do not exist in all Iterations. The internal Iteration layouts should be homogeneous.
+        If you need this feature, please contact the openPMD-api developers; implementing this is currently not a priority.
+        Datasets that do not exist in all steps will be skipped at read time (with an error).
+     3. Datasets with changing extents are supported.
 
 * **Read/Write mode**: Creates a new Series if not existing, otherwise opens an existing Series for reading and writing.
   New datasets and iterations will be inserted as needed.

examples/5_write_parallel.py

@@ -56,8 +56,7 @@
     # in streaming setups, e.g. an iteration cannot be opened again once
     # it has been closed.
     # `Series.iterations` can be directly accessed in random-access workflows.
-    series.iterations[1].open()
-    mymesh = series.iterations[1]. \
+    mymesh = series.write_iterations()[1]. \
         meshes["mymesh"]
 
     # example 1D domain decomposition in first index
@@ -92,7 +91,7 @@
     # The iteration can be closed in order to help free up resources.
     # The iteration's content will be flushed automatically.
     # An iteration once closed cannot (yet) be reopened.
-    series.iterations[1].close()
+    series.write_iterations()[1].close()
 
     if 0 == comm.rank:
         print("Dataset content has been fully written to disk")

include/openPMD/Datatype.hpp

@@ -513,7 +513,7 @@ inline bool isFloatingPoint(Datatype d)
  * @param d Datatype to test
  * @return true if complex floating point, otherwise false
  */
-inline bool isComplexFloatingPoint(Datatype d)
+constexpr inline bool isComplexFloatingPoint(Datatype d)
 {
     using DT = Datatype;
 
@@ -554,7 +554,7 @@ inline bool isFloatingPoint()
  * @return true if complex floating point, otherwise false
  */
 template <typename T>
-inline bool isComplexFloatingPoint()
+constexpr inline bool isComplexFloatingPoint()
 {
     Datatype dtype = determineDatatype<T>();
 

include/openPMD/IO/ADIOS/ADIOS2Auxiliary.hpp

@@ -97,6 +97,8 @@ namespace adios_defaults
     constexpr const_str str_usesteps = "usesteps";
     constexpr const_str str_flushtarget = "preferred_flush_target";
     constexpr const_str str_usesstepsAttribute = "__openPMD_internal/useSteps";
+    constexpr const_str str_useModifiableAttributes =
+        "__openPMD_internal/useModifiableAttributes";
     constexpr const_str str_adios2Schema =
         "__openPMD_internal/openPMD2_adios2_schema";
     constexpr const_str str_isBoolean = "__is_boolean__";

include/openPMD/IO/ADIOS/ADIOS2File.hpp

@@ -79,7 +79,8 @@ struct DatasetReader
         BufferedGet &bp,
         adios2::IO &IO,
         adios2::Engine &engine,
-        std::string const &fileName);
+        std::string const &fileName,
+        std::optional<size_t> stepSelection);
 
     static constexpr char const *errorMsg = "ADIOS2: readDataset()";
 };
@@ -412,6 +413,8 @@ class ADIOS2File
     StreamStatus streamStatus = StreamStatus::OutsideOfStep;
 
     size_t currentStep();
+    void setStepSelection(std::optional<size_t>);
+    [[nodiscard]] std::optional<size_t> stepSelection() const;
 
 private:
     ADIOS2IOHandlerImpl *m_impl;
@@ -420,8 +423,11 @@ class ADIOS2File
     /*
      * Not all engines support the CurrentStep() call, so we have to
      * implement this manually.
+     * Note: We don't use a std::optional<size_t> here since the currentStep
+     * is always being counted.
      */
     size_t m_currentStep = 0;
+    bool useStepSelection = false;
 
     /*
      * ADIOS2 does not give direct access to its internal attribute and

include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp

@@ -23,6 +23,7 @@
 #include "openPMD/Error.hpp"
 #include "openPMD/IO/ADIOS/ADIOS2Auxiliary.hpp"
 #include "openPMD/IO/ADIOS/ADIOS2FilePosition.hpp"
+#include "openPMD/IO/ADIOS/macros.hpp"
 #include "openPMD/IO/AbstractIOHandler.hpp"
 #include "openPMD/IO/AbstractIOHandlerImpl.hpp"
 #include "openPMD/IO/AbstractIOHandlerImplCommon.hpp"
@@ -190,6 +191,9 @@ class ADIOS2IOHandlerImpl
 
     void readAttribute(Writable *, Parameter<Operation::READ_ATT> &) override;
 
+    void readAttributeAllsteps(
+        Writable *, Parameter<Operation::READ_ATT_ALLSTEPS> &) override;
+
     void listPaths(Writable *, Parameter<Operation::LIST_PATHS> &) override;
 
     void
@@ -431,7 +435,8 @@ class ADIOS2IOHandlerImpl
         Offset const &offset,
         Extent const &extent,
         adios2::IO &IO,
-        std::string const &varName)
+        std::string const &varName,
+        std::optional<size_t> stepSelection)
     {
         {
             auto requiredType = adios2::GetType<T>();
@@ -458,6 +463,10 @@ class ADIOS2IOHandlerImpl
             throw std::runtime_error(
                 "[ADIOS2] Internal error: Failed opening ADIOS2 variable.");
         }
+        if (stepSelection.has_value())
+        {
+            var.SetStepSelection({*stepSelection, 1});
+        }
         // TODO leave this check to ADIOS?
         adios2::Dims shape = var.Shape();
         auto actualDim = shape.size();
@@ -533,11 +542,8 @@ namespace detail
     struct AttributeReader
     {
         template <typename T>
-        static Datatype call(
-            ADIOS2IOHandlerImpl &,
-            adios2::IO &IO,
-            std::string name,
-            Attribute::resource &resource);
+        static Datatype
+        call(adios2::IO &IO, std::string name, Attribute::resource &resource);
 
         template <int n, typename... Params>
         static Datatype call(Params &&...);
@@ -562,7 +568,8 @@ namespace detail
             ADIOS2IOHandlerImpl *impl,
             InvalidatableFile const &,
             std::string const &varName,
-            Parameter<Operation::OPEN_DATASET> &parameters);
+            Parameter<Operation::OPEN_DATASET> &parameters,
+            std::optional<size_t> stepSelection);
 
         static constexpr char const *errorMsg = "ADIOS2: openDataset()";
     };
@@ -854,6 +861,11 @@ class ADIOS2IOHandler : public AbstractIOHandler
         return "ADIOS2";
     }
 
+    bool fullSupportForVariableBasedEncoding() const override
+    {
+        return true;
+    }
+
     std::future<void> flush(internal::ParsedFlushParams &) override;
 }; // ADIOS2IOHandler
 } // namespace openPMD

include/openPMD/IO/AbstractIOHandler.hpp

@@ -250,6 +250,7 @@ class AbstractIOHandler
 
     /** The currently used backend */
     virtual std::string backendName() const = 0;
+    virtual bool fullSupportForVariableBasedEncoding() const;
 
     std::string directory;
     /*

include/openPMD/IO/AbstractIOHandlerImpl.hpp

@@ -72,6 +72,13 @@ class AbstractIOHandlerImpl
      * IO actions up to the point of closing a step must be performed now.
      *
      * The advance mode is determined by parameters.mode.
+     * parameters.mode has type std::variant<AdvanceMode, StepSelection>:
+     *
+     * 1. AdvanceMode is for processing steps sequentially. In this case, a step
+     *    is either begun or closed.
+     * 2. StepSelection is for random-accessing steps. A target step number is
+     *    specified.
+     *
      * The return status code shall be stored as parameters.status.
      */
     virtual void advance(Writable *, Parameter<Operation::ADVANCE> &parameters)
@@ -360,6 +367,24 @@ class AbstractIOHandlerImpl
      */
     virtual void
     readAttribute(Writable *, Parameter<Operation::READ_ATT> &) = 0;
+    /** Collective task to read modifiable attributes over steps.
+     *
+     * Has a default implementation for backends that do not support steps;
+     * here, the task is relayed to normal READ_ATT.
+     * This task is key for implementing the preparsing logic needed in
+     * random-access read mode for variable-encoded ADIOS2 files.
+     * adios2::Mode::ReadRandomAccess does not support modifiable attributes,
+     * so this task will instead quickly open the file's metadata in
+     * adios2::Mode::Read, go through all its steps and register the attribute
+     * values. Expensive and collective operation, run only once at startup.
+     * Absolutely necessary for reading /data/snapshot.
+     * Necessary (but not yet used) for having correct values in attributes
+     * such as /data/time.
+     * In future: Let this task preparse the entirety of all modifiable
+     * attributes.
+     */
+    virtual void readAttributeAllsteps(
+        Writable *, Parameter<Operation::READ_ATT_ALLSTEPS> &);
     /** List all paths/sub-groups inside a group, non-recursively.
      *
      * The operation should fail if the Writable was not marked written.