add flag append_mode, complete documentation, add tests accordingly

Author: atomprobe-tc

docs/index.md

@@ -69,7 +69,7 @@ We are offering a small guide to getting started with NeXus, `pynxtools`, and NO
 - [Data conversion in `pynxtools`](learn/pynxtools/dataconverter-and-readers.md)
 - [Validation of NeXus files](learn/pynxtools/nexus-validation.md)
 - [The `MultiFormatReader` as a reader superclass](learn/pynxtools/multi-format-reader.md)
-- [Append mode `dataconverter`](learn/pynxtools/dataconverter-append-mode.md)
+- [Append mode for the dataconverter](learn/pynxtools/dataconverter-append-mode.md)
 
 </div>
 <div markdown="block">
@@ -103,4 +103,4 @@ For questions or suggestions:
 
 <h2>Project and community</h2>
 
-The work is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - [460197019 (FAIRmat)](https://gepris.dfg.de/gepris/projekt/460197019?language=en).
+The work is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - [460197019 (FAIRmat)](https://gepris.dfg.de/gepris/projekt/460197019?language=en).

docs/learn/pynxtools/dataconverter-append-mode.md

@@ -3,45 +3,94 @@
 There are cases where users wish to compose a NeXus/HDF5 file with data from multiple sources. Typical examples include:
 
 - A file should contain multiple `NXentry` instances where each instance applies a different application definition.
-- Content under `NXentry` instances is composed from running a specific pynxtools parser plugin plus additional content
-  that is injected via software other than `pynxtools` or not even software that is written in Python.
+- Content within `NXentry` instances is generated by executing a specific `pynxtools` reader plugin and may be supplemented
+  with additional data provided by external software components, including those implemented outside of Python.
 
 Enabling such use cases while minimizing data copying is the idea behind the append mode of the dataconverter. It is activated by
-passing the `--append` flag during [command line invocation](../../tutorial/converting-data-to-nexus.md).
+passing the `--append` flag during command line invocation (see [Tutorial -> Converting your research data to NeXus](../../tutorial/converting-data-to-nexus.md).
+
+Taking this tutorial and its NXxps case study as an example. It composes the HDF5 file with content from two input files
+the `EX439_S718_Au.sle` with proprietary formatting and the `eln_data_sle.yaml`, a NOMAD-specific metadata exchange file.
+Instead of running the tutorial with passing both input in one go, one could first add process only the proprietary file
+(without using `--append`) and thereafter process the YAML file (with using `--append`). The minimal command line call
+reads as follows.
+
+```
+dataconverter EX439_S718_Au.sle --reader xps --nxdl NXxps --output Au_25_mbar_O2_no_align.nxs
+dataconverter eln_data_sle.yaml --reader xps --nxdl NXxps --append --output Au_25_mbar_O2_no_align.nxs
+```
+
+When processing both the `*.sle` and `*.yaml` file in one call, adding `--append` has no effect, i.e.,
+`pynxtools` proceeds as if `--append` is absent but mind that adding the flag deactivates the verification.
+
+Users who wish to use a `params.yaml` parameters file, like it is shown in the tutorial,
+should add the `append` flag like this:
+
+```
+dataconverter:
+  reader: xps
+  nxdl: NXxps
+  input-file:
+    - EX439_S718_Au.sle
+    - eln_data_sle.yaml
+  output: Au_25_mbar_O2_no_align.nxs
+  append: True
+```
+
+Users who wish to call the dataconverter as a step in other Python code or Jupyter Notebooks may find
+the following variation and code a useful snippet to include in their batch pipeline:
+
+```
+from pynxtools.dataconverter.convert import convert
+
+# file "Au_25_mbar_O2_no_align.nxs" exists already, e.g., when it
+# was instantiated with "EX439_S718_Au.sle" as mentioned above
+
+_ = convert(
+    input_file=("eln_data_sle.yaml"),
+    reader="xps",
+    nxdl="NXxps",
+    append=True,
+    output="Au_25_mbar_O2_no_align.nxs",
+)
+
+# modify tuple[str] input_file to include the actual files you wish to convert
+# modify output: str to customize output file path and name
+```
 
 ## Possibilities and limitations
 
-**The append mode must not be understood as a functionality that allows an overwriting of existent data.**
+**The append mode is not a functionality that allows an overwriting of existent data!**
 We are convinced that written data should be immutable. Therefore, using the append mode demands to accept the following assumptions:
 
 - Only groups, datasets, or attributes not yet existent can be added when in append mode.
   The implementation catches attempts of overwriting existent HDF5 objects,
   emitting respective logging messages.
 - When in append mode, the internal validation of the `template` dictionary is switched off,
-  irrespective if `--skip-verify` is passed or not.
-  Instead, users should validate [the HDF5 file](../../how-tos/pynxtools/validate-nexus-files.md) when having the file compositing completed.
+  irrespective if `--skip-verify` is passed or not. Instead, users should validate the HDF5 file
+  (see [How-tos -> pynxtools -> Validation of NeXus files](../../how-tos/pynxtools/validate-nexus-files.md) after they have completed the file compositing.
 - The HDF5 library's functionality to reshape existent HDF5 datasets is not supported by `pynxtools`.
 
 ## Interpreting root level attributes
 
 Note that `pynxtools` sets several attributes at the root level of a NeXus/HDF5 file. These values are defined by whichever tool writes them first.
-A subsequent writing to the HDF5 file in append mode does not modify these. This makes the interpretation of the following attributes ambiguous
-`NeXus_repository`, `NeXus_release`, `HDF5_Version`, `h5py_version`, `creator`, `creator_version`, `file_time` and `file_update_time`.
+A subsequent writing to the HDF5 file in append mode does not modify these. This makes the interpretation of the following attributes ambiguous:
+`NeXus_repository`, `NeXus_release`, `HDF5_Version`, `h5py_version`, `creator`, `creator_version`, `file_time`, and `file_update_time`.
 
 When in append mode, `pynxtools` adds the root level attribute `append_mode = "True"` which flags the file as an artifact that was composed
-from at least one pynxtools tool running in append mode. Note that the absence of this flag does not guarantee that the file was written
-by `pynxtools` or its plugins, as also other software could have written the NeXus file.
+from at least one `pynxtools` tool or reader plugin running in append mode. Note that the absence of this flag does not guarantee that the file
+was written by `pynxtools` or its plugins, as also other software could have written the NeXus file.
 
-Until the NeXus standard allows users to link or define these attributes at the HDF5 object level, i.e. for groups, datasets, and attributes, separately,
-we advise to no mix tools that write content that adheres to different versions of the NeXus definitions. Note that the `validate` functionality
-of `pynxtools` can currently not detect which objects within an HDF5 file were written with which NeXus or tool version. The validation concludes from
-the combination of the `ENTRY/definition`, `ENTRY/definition/@version`, and `/@NeXus_version` attributes.
+Until the NeXus standard allows users to link or define these attributes at the HDF5 object level, i.e., for groups, datasets, and attributes, separately,
+we advise not to mix tools that write content that adheres to different versions of the NeXus definitions. Note that the `validate` functionality
+of `pynxtools` does not provide a mechanism to determine which specific NeXus or tool version was used to generate individual objects within an HDF5 file.
+The validation concludes from the combination of the `ENTRY/definition`, `ENTRY/definition/@version`, and `/@NeXus_version` attributes.
 
 ## Time-stamped HDF5 objects
 
 Note that the HDF5 library has the low-level feature to timestamp individual HDF5 objects. By default though, this feature is deactivated
 as per decision of the HDF5 Consortium. The choice was made to prevent that changing timestamp values change the hash of the entire file content.
 Note that the `pynxtools-em` plugin includes a [`hfive_base` parser](https://github.com/FAIRmat-NFDI/pynxtools-em/blob/main/src/pynxtools_em/parsers/hfive_base.py)
-that can compute hashes from the content of individual HDF5 objects. Users are advised to blacklist timestamp attributes like `file_time`, and `file_update_time`
+that can compute hashes from the content of individual HDF5 objects. Users are advised to blacklist timestamp attributes like `file_time` and `file_update_time`
 when comparing the binary content of two HDF5 files using this parser. 
 

src/pynxtools/dataconverter/convert.py

@@ -344,7 +344,9 @@ def main_cli():
     "--skip-verify",
     is_flag=True,
     default=False,
-    help="Skips the verification routine during conversion.",
+    help="Skips the verification routine during conversion."
+    "When --append is used the verification is always skipped"
+    "irrespective if --skip-verify is used or not.",
 )
 @click.option(
     "--mapping",

src/pynxtools/dataconverter/writer.py

@@ -225,7 +225,7 @@ def __init__(
         self.data = data
         self.nxdl_f_path = nxdl_f_path
         self.output_path = output_path
-        self.output_nexus = h5py.File(self.output_path, "r+" if append else "w")
+        self.output_nexus = h5py.File(self.output_path, "a" if append else "w")
         # using "r+" or "a" allow resizing a dataset that uses chunked data storage layout
         # we currently do not implement this resizing though
         # create_{group,dataset} with an existent name throws a ValueError