FAIRmat-NFDI · lukaspie · Aug 21, 2025 · May 17, 2024 · May 21, 2024 · Jun 4, 2024
diff --git a/dev-requirements.txt b/dev-requirements.txt
@@ -8,6 +8,8 @@ babel==2.17.0
     # via mkdocs-material
 backrefs==5.9
     # via mkdocs-material
+cachetools==6.1.0
+    # via pynxtools (pyproject.toml)
 certifi==2025.7.14
     # via requests
 cfgv==3.4.0

diff --git a/docs/learn/pynxtools/nexus-validation.md b/docs/learn/pynxtools/nexus-validation.md
@@ -1,23 +1,43 @@
-# Validation of NeXus file
+# Validation of NeXus files
 
 !!! info "This page is intended to give more information about the validation tools that are part of `pynxtools`. Please also have a look at our comprehensive [how-to guide](../../how-tos/pynxtools/validate-nexus-file.md) on NeXus validation."
 
-One of the main advantages of using pynxtools is that it comes with its own validation tools. That is, it can be used to validate that a given NeXus/HDF5 file is compliant with a NeXus application definition.
+One of the main advantages of using `pynxtools` is that it comes with its own validation tools. That is, it can be used to validate that a given NeXus/HDF5 file is compliant with a NeXus application definition.
 
 ## As part of the dataconverter
 
-During [data conversion](./dataconverter-and-readers.md), before writing the HDF5 file, the data is first checked against the provided application definition.
+During [data conversion](./dataconverter-and-readers.md) within `pynxtools`, before writing the HDF5 file, the data is first checked against the provided application definition.
 
-<!--## verify-nexus: Testing existing NeXus/HDF5 files
-This CLI tool can be used to validate _existing_ HDF5 files that claim to be NeXus-compliant. See [here](reference/cli-api.md#verify-nexus) for the API documentation.-->
+## validate_nexus: Validate existing NeXus/HDF5 files
+
+While we encourage NeXus users to convert their data using the `pynxtools` data converter, we also realize that a lot of NeXus files are created using other applications. For such use cases,
+`pynxtools` provides a standalone validator (called `validate_nexus`). This CLI tool can be used to validate _existing_ HDF5 files against the NeXus application definition they claim to be comply with. Read more in the [API documentation](../../reference/cli-api.md#validate_nexus).
+
+The following example dataset can be used to test the `validate_nexus` module:
+
+[201805_WSe2_arpes.nxs](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/data/201805_WSe2_arpes.nxs){:target="_blank" .md-button }
+
+This is an angular-resolved photoelectron spectroscopy (ARPES) dataset that is formatted according to the [`NXarpes` application definition](https://manual.nexusformat.org/classes/applications/NXarpes.html#nxarpes).
+
+If you have `pynxtools` installed, you can call the validator on this file using the command
+
+```bash
+validate_nexus 201805_WSe2_arpes.nxs
+```
+
+You will see some warning messages that will give you an impression of the kind of messages the validator tool provides.
 
 ## read_nexus: NeXus file reader and debugger
 
 This utility outputs a debug log for a given NeXus file by annotating the data and metadata entries with the schema definitions from the respective NeXus base classes and application definitions to which the file refers to. See [here](../../reference/cli-api.md#nexus-file-validation) for the API documentation.
 
-The following example dataset can be used to test the `read_nexus` module: [src/pynxtools/data/201805_WSe2_arpes.nxs](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/data/201805_WSe2_arpes.nxs). This is an angular-resolved photoelectron spectroscopy (ARPES) dataset that is formatted according to the [NXarpes application definition of NeXus](https://manual.nexusformat.org/classes/applications/NXarpes.html#nxarpes).
+If you have `pynxtools` installed, you can call the tool on the file mentioned above using the command
+
+```bash
+read_nexus 201805_WSe2_arpes.nxs
+```
 
-!!! info "Using a different set of NeXus definitions"
+??? info "Using a different set of NeXus definitions"
 
     The environment variable "NEXUS_DEF_PATH" can be set to a directory which contains the NeXus definitions as NXDL XML files. If this environment variable is not defined, the module will use the definitions in its bundle (see `src/pynxtools/definitions`)._
 
@@ -26,7 +46,7 @@ The following example dataset can be used to test the `read_nexus` module: [src/
     export 'NEXUS_DEF_PATH'=<folder_path_that_contains_nexus_defs>
     ```
 
-!!! info "A note to Windows users"
+??? info "A note to Windows users"
 
     If you run `read_nexus` from `git bash`, you need to set the environmental variable
     `MSYS_NO_PATHCONV` to avoid the [path translation in Windows Git MSys](https://stackoverflow.com/questions/7250130/how-to-stop-mingw-and-msys-from-mangling-path-names-given-at-the-command-line#34386471).
@@ -40,10 +60,10 @@ The following example dataset can be used to test the `read_nexus` module: [src/
 
 ## Other approaches (not part of pynxtools)
 
-Aside from the tools we developed within FAIRmat, the [official NeXus website](https://manual.nexusformat.org/validation.html) lists additional programs for the validation of NeXus files:
+Aside from the tools we develop within FAIRmat, the [official NeXus website](https://manual.nexusformat.org/validation.html) lists additional programs for the validation of NeXus files:
 
 1. [cnxvalidate: NeXus validation tool written in C](https://github.com/nexusformat/cnxvalidate)
 2. [punx: Python Utilities for NeXus HDF5 files](https://github.com/prjemian/punx)
 3. [nexpy/nxvalidate: A python API for validating NeXus file](https://github.com/nexpy/nxvalidate)
 
-We will not discuss the details of these programs here, but you can find some information about the in the how-to guide linked above.
+We will not discuss the details of these programs here, but you can find some information about them when following these links.
diff --git a/docs/reference/cli-api.md b/docs/reference/cli-api.md
@@ -15,13 +15,16 @@ Note that simply calling `dataconverter` defaults to `dataconverter convert`.
     :list_subcommands: True
 
 ## NeXus file validation
-<!-- ::: mkdocs-click
-    :module: "pynxtools.dataconverter.verify
-    :command: verify_nexus
-    :prog_name: verify_nexus
-    :depth: 1
+
+::: mkdocs-click
+    :module: pynxtools.dataconverter.validate_file
+    :command: validate_cli
+    :prog_name: validate_nexus
+    :depth: 2
     :style: table
-    :list_subcommands: True -->
+    :list_subcommands: True
+
+## NeXus annotator
 
 ::: mkdocs-click
     :module: pynxtools.nexus.nexus

diff --git a/pyproject.toml b/pyproject.toml
@@ -36,6 +36,7 @@ dependencies = [
     "toposort>=1.10.0",
     "anytree",
     "pint",
+    "cachetools",
 ]
 
 [project.urls]
@@ -108,6 +109,7 @@ simple_nexus_example = "pynxtools.nomad.entrypoints:simple_nexus_example"
 read_nexus = "pynxtools.nexus.nexus:main"
 dataconverter = "pynxtools.dataconverter.convert:main_cli"
 generate_eln = "pynxtools.eln_mapper.eln_mapper:get_eln"
+validate_nexus = "pynxtools.dataconverter.validate_file:validate_cli"
 
 [tool.setuptools.package-data]
 pynxtools = ["definitions/**/*.xml", "definitions/**/*.xsd"]

diff --git a/src/pynxtools/data/NXtest.nxdl.xml b/src/pynxtools/data/NXtest.nxdl.xml
@@ -45,14 +45,14 @@
         </group>
         <group type="NXdata" name="specified_group" nameType="specified">
             <doc>A group with a name and nameType="specified".</doc>
-            <field name="specified_field" nameType="specified" type="NX_FLOAT" units="NX_ANY">
+            <field name="specified_field" nameType="specified" optional="true" type="NX_FLOAT" units="NX_ANY">
                 <attribute name="specified_attr_in_field" nameType="specified"/>
             </field>
             <attribute name="specified_attr"/>
         </group>
         <group type="NXdata" name="any_groupGROUP" nameType="any">
             <doc>A group with a name and nameType="any".</doc>
-            <field name="any_fieldFIELD" nameType="any" optional="true" type="NX_FLOAT" units="NX_ANY">
+            <field name="any_fieldFIELD" nameType="any" type="NX_FLOAT" units="NX_ANY">
                 <attribute name="any_attrATTR_in_field" nameType="any"/>
             </field>
             <attribute name="any_attrATTR" nameType="any"/>

diff --git a/src/pynxtools/dataconverter/convert.py b/src/pynxtools/dataconverter/convert.py
@@ -216,8 +216,8 @@ def convert(
     fair : bool, default False
         If True, a warning is given that there are undocumented paths
         in the template.
-    undocumented : bool, default False
-        If True, an undocumented warning is given.
+    ignore_undocumented : bool, default False
+        If True, all undocumented items are ignored in the validation.
     skip_verify: bool, default False
         Skips verification routine if set to True
 
@@ -324,7 +324,7 @@ def main_cli():
     "--ignore-undocumented",
     is_flag=True,
     default=False,
-    help="Ignore all undocumented fields during validation.",
+    help="Ignore all undocumented concepts during validation.",
 )
 @click.option(
     "--fail",