add flag append_mode, complete documentation, add tests accordingly

Author: atomprobe-tc

.cspell/custom-dictionary.txt

@@ -116,6 +116,7 @@ hashvalue
 hdfdict
 hdfgroup
 hdfobject
+hfive
 hixu
 hypothes
 idname

.pre-commit-config.yaml

@@ -11,6 +11,8 @@ repos:
     rev: v1.19.1
     hooks:
       - id: mypy  # static type checking
+        additional_dependencies:
+          - types-PyYAML
 
   - repo: https://github.com/asottile/pyupgrade
     rev: v3.21.1

docs/learn/pynxtools/dataconverter-append-mode.md

@@ -4,19 +4,44 @@ There are cases where users wish to compose a NeXus/HDF5 file with data from mul
 
 - A file should contain multiple `NXentry` instances where each instance applies a different application definition.
 - Content under `NXentry` instances is composed from running a specific pynxtools parser plugin plus additional content
-  that is injected via software that might not be written in Python.
+  that is injected via software other than `pynxtools` or not even software that is written in Python.
 
 Enabling such use cases while minimizing data copying is the idea behind the append mode of the dataconverter. It is activated by
 passing the `--append` flag during [command line invocation](../../tutorial/converting-data-to-nexus.md).
 
-**The append mode must not be understood as a functionality that allows overwriting data.** Convinced that data should be immutable,
-once generated, the append mode feature works with the following key assumptions:
+## Possibilities and limitations
+
+**The append mode must not be understood as a functionality that allows an overwriting of existent data.**
+We are convinced that written data should be immutable. Therefore, using the append mode demands to accept the following assumptions:
 
 - Only groups, datasets, or attributes not yet existent can be added when in append mode.
-  The implementation catches attempts of overwriting existent such type of HDF5 objects,
+  The implementation catches attempts of overwriting existent HDF5 objects,
   emitting respective logging messages.
 - When in append mode, the internal validation of the `template` dictionary is switched off,
   irrespective if `--skip-verify` is passed or not.
-  Instead, users should validate [the HDF5 file a posteriori](../../how-tos/pynxtools/validate-nexus-files.md).
-- Despite the HDF5 library offers the functionality, a reshaping of HDF5 datasets is not supported.
+  Instead, users should validate [the HDF5 file](../../how-tos/pynxtools/validate-nexus-files.md) when having the file compositing completed.
+- The HDF5 library's functionality to reshape existent HDF5 datasets is not supported by `pynxtools`.
+
+## Interpreting root level attributes
+
+Note that `pynxtools` sets several attributes at the root level of a NeXus/HDF5 file. These values are defined by whichever tool writes them first.
+A subsequent writing to the HDF5 file in append mode does not modify these. This makes the interpretation of the following attributes ambiguous
+`NeXus_repository`, `NeXus_release`, `HDF5_Version`, `h5py_version`, `creator`, `creator_version`, `file_time` and `file_update_time`.
+
+When in append mode, `pynxtools` adds the root level attribute `append_mode = "True"` which flags the file as an artifact that was composed
+from at least one pynxtools tool running in append mode. Note that the absence of this flag does not guarantee that the file was written
+by `pynxtools` or its plugins, as also other software could have written the NeXus file.
+
+Until the NeXus standard allows users to link or define these attributes at the HDF5 object level, i.e. for groups, datasets, and attributes, separately,
+we advise to no mix tools that write content that adheres to different versions of the NeXus definitions. Note that the `validate` functionality
+of `pynxtools` can currently not detect which objects within an HDF5 file were written with which NeXus or tool version. The validation concludes from
+the combination of the `ENTRY/definition`, `ENTRY/definition/@version`, and `/@NeXus_version` attributes.
+
+## Time-stamped HDF5 objects
+
+Note that the HDF5 library has the low-level feature to timestamp individual HDF5 objects. By default though, this feature is deactivated
+as per decision of the HDF5 Consortium. The choice was made to prevent that changing timestamp values change the hash of the entire file content.
+Note that the `pynxtools-em` plugin includes a [`hfive_base` parser](https://github.com/FAIRmat-NFDI/pynxtools-em/blob/main/src/pynxtools_em/parsers/hfive_base.py)
+that can compute hashes from the content of individual HDF5 objects. Users are advised to blacklist timestamp attributes like `file_time`, and `file_update_time`
+when comparing the binary content of two HDF5 files using this parser. 
 

src/pynxtools/dataconverter/convert.py

@@ -227,7 +227,11 @@ def convert(
         **kwargs,
     )
 
-    helpers.add_default_root_attributes(data=data, filename=os.path.basename(output))
+    helpers.add_default_root_attributes(
+        data=data,
+        filename=os.path.basename(output),
+        append=True if "append" in kwargs else False,
+    )
     Writer(
         data=data,
         nxdl_f_path=nxdl_f_path,

src/pynxtools/dataconverter/helpers.py

@@ -1259,7 +1259,7 @@ def convert_to_hill(atoms_typ):
     return atom_list + list(atoms_typ)
 
 
-def add_default_root_attributes(data, filename):
+def add_default_root_attributes(data, filename, append: bool = False):
     """
     Takes a dict/Template and adds NXroot fields/attributes that are inherently available
     """
@@ -1284,6 +1284,8 @@ def update_and_warn(key: str, value: str):
     update_and_warn("/@NeXus_release", get_nexus_version())
     update_and_warn("/@HDF5_Version", ".".join(map(str, h5py.h5.get_libversion())))
     update_and_warn("/@h5py_version", h5py.__version__)
+    if append:
+        update_and_warn("/@append_mode", "True")
 
 
 def write_nexus_def_to_entry(data, entry_name: str, nxdl_def: str):

tests/dataconverter/test_writer.py

@@ -25,6 +25,7 @@
 import pytest
 
 from pynxtools.dataconverter.exceptions import InvalidDictProvided
+from pynxtools.dataconverter.helpers import add_default_root_attributes
 from pynxtools.dataconverter.template import Template
 from pynxtools.dataconverter.writer import Writer
 
@@ -38,10 +39,12 @@
 @pytest.fixture(name="writer")
 def fixture_writer(filled_test_data, tmp_path):
     """pytest fixture to setup Writer object to be used by tests with dummy data."""
+    output_file_path = os.path.join(tmp_path, "test.nxs")
+    add_default_root_attributes(filled_test_data, filename=output_file_path)
     writer = Writer(
         filled_test_data,
         os.path.join(os.getcwd(), "src", "pynxtools", "data", "NXtest.nxdl.xml"),
-        os.path.join(tmp_path, "test.nxs"),
+        output_file_path,
     )
     yield writer
     del writer
@@ -55,10 +58,11 @@ def test_init(writer):
 def test_write(writer):
     """Test for the Writer's write function. Checks whether entries given above get written out."""
     writer.write()
-    test_nxs = h5py.File(writer.output_path, "r")
-    assert test_nxs["/my_entry/nxodd_name/int_value"][()] == 2
-    assert test_nxs["/my_entry/nxodd_name/int_value"].attrs["units"] == "eV"
-    assert test_nxs["/my_entry/nxodd_name/posint_value"].shape == (3,)  # pylint: disable=no-member
+    with h5py.File(writer.output_path, "r") as test_nxs:
+        assert "/append_mode" not in test_nxs
+        assert test_nxs["/my_entry/nxodd_name/int_value"][()] == 2
+        assert test_nxs["/my_entry/nxodd_name/int_value"].attrs["units"] == "eV"
+        assert test_nxs["/my_entry/nxodd_name/posint_value"].shape == (3,)  # pylint: disable=no-member
 
 
 def test_write_link(writer):
@@ -129,6 +133,7 @@ def fixture_normal_write_then_attempt_append(writer):
     template["/ENTRY[my_entry]/definition/@version"] = "2.4.6"
     template["/ENTRY[my_entry]/required_group/description"] = "An example description"
     template["/already/existing_value"] = np.zeros((125000,), dtype=np.float64)
+    add_default_root_attributes(template, filename=writer.output_path, append=True)
     overwrite = Writer(
         template,
         os.path.join(os.getcwd(), "src", "pynxtools", "data", "NXtest.nxdl.xml"),
@@ -143,13 +148,25 @@ def test_overwrite(writer_overwrite, caplog):
     """Test whether append is correctly working for the writer."""
     writer_overwrite.write()
 
+    with h5py.File(writer_overwrite.output_path, "r") as test_nxs:
+        assert "append_mode" in test_nxs["/"].attrs
+        assert test_nxs["/"].attrs["append_mode"] == "True"
+
     with caplog.at_level(logging.INFO):
         observed_infos = [
             r.getMessage() for r in caplog.records if r.levelno == logging.INFO
         ]
 
         prefix = "Prevented the overwriting of"
         expected_infos = [
+            f"{prefix} attribute /@HDF5_Version",
+            f"{prefix} attribute /@NX_class",
+            f"{prefix} attribute /@NeXus_release",
+            f"{prefix} attribute /@NeXus_repository",
+            f"{prefix} attribute /@file_name",
+            f"{prefix} attribute /@file_time",
+            f"{prefix} attribute /@file_update_time",
+            f"{prefix} attribute /@h5py_version",
             f"{prefix} attribute /ENTRY[my_entry]/@NX_class",
             f"{prefix} dataset /ENTRY[my_entry]/definition",
             f"{prefix} dataset /ENTRY[my_entry]/required_group/description",