Merge pull request #402 from FAIRmat-NFDI/nexus_validation_tutorial

seperated validation and creation tutorial,
fixed several typos and languange
restuctured links and headings for TOC sidebar

Author: RonHildebrandt

docs/how-tos/attachments/51dc82f9f0f5ec2f-image.png

@@ -0,0 +1,246 @@
+# Use python to create NeXus files
+
+__The goal__
+
+Use python to create a NeXus file (.nxs) by hardcoding via the python package `h5py`. NeXus files can also be created by our software [`pynxtools`](https://github.com/FAIRmat-NFDI/pynxtools) automatically, but ONLY IF a reader for the specific device/instrument/data-structure exists. This How-To is intended as easy access to FAIR data structures _via_ NeXus. For static-datastructures (i.e., always the same type of standard measurement) or one-time examples (small data publications), this may provide a feasible solution. For large scaled automated file processing, storage, and validation, it is advisable to use [`pynxtools`](https://github.com/FAIRmat-NFDI/pynxtools) and its measurement method specific [plugins](../reference/plugins.md)
+
+You can find the necessary file downloads [here](https://zenodo.org/records/13373909).
+
+
+
+## Make NeXus file by python
+
+Install `h5py` via `pip`:
+```console
+`pip install h5py`
+```
+
+Then you can create a NeXus file by the python script called [h5py_nexus_file_creation.py](https://zenodo.org/records/13373909/files/h5py_nexus_file_creation.py?download=1).
+
+```
+# Import h5py, to write an hdf5 file
+import h5py
+
+# create a h5py file in writing mode with given name "NXopt_minimal_example", file extension "nxs"
+f = h5py.File("NXopt_minimal_example.nxs", "w")
+
+# there are only 3 fundamental objects: >group<, >attribute< and >datafield<.
+
+
+# create a >group< called "entry"
+f.create_group('/entry')
+
+# assign the >group< called "entry" an >attribute<
+# The attribute is "NX_class"(a NeXus class) with the value of this class is "NXentry"
+f['/entry'].attrs['NX_class'] = 'NXentry'
+
+# create >datafield< called "definition" inside the entry, and assign it the value "NXoptical_spectroscopy"
+# This field is important, as it is used in validation process to identify the NeXus definition.
+f['/entry/definition'] = 'NXoptical_spectroscopy'
+```
+
+This proves a starting point of the NeXus file. We will go through these functions in the following.
+
+
+
+## Add NeXus concepts by python
+
+Go to [FAIRmat NeXus definitions](<https://fairmat-nfdi.github.io/nexus_definitions/index.html#>)
+
+Scroll down until you see the search box named "Quick search".
+
+Type "NXoptical" and press start the search.
+
+You see several search results, select the one with is named "NXoptical\_spectroscopy".
+
+Then you are (ideally) on this page: [NXoptical_spectroscopy NeXus definition](<https://fairmat-nfdi.github.io/nexus_definitions/classes/contributed_definitions/NXoptical_spectroscopy.html>)
+
+You see a tree-like structure of the NeXus definition NXoptical\_spectrosocopy with several tree nodes: Status, Description, Symbols, Groups\_cited, Structure. For now, only the part in Structure is of interest. This contains the information which has to be written in the python code to add fields/groups/attributes to the NeXus file.
+
+Use your browser search (CRTL+F) and search for "required". Ideally, your browser highlights all concepts which are required. You have to add those to the python script to extend your created .nxs file. (Which fields/groups/attributes are "required" was defined by the respective scientific community, to ensure that the data serves the FAIR principles.)
+
+In the following, it will be shown how the python script has to be extended for the three fundamental objects:
+
+1. Attribute
+
+2. Datafield
+
+3. Group
+
+
+
+
+
+### Adding an attribute
+
+Search for the first concept/object in the NeXus file which is not created yet. It is:
+
+**@version**: (required) [NX\_CHAR](<https://fairmat-nfdi.github.io/nexus_definitions/nxdl-types.html#nx-char>) [⤆](<https://fairmat-nfdi.github.io/nexus_definitions/classes/base_classes/NXentry.html#nxentry-definition-version-attribute>)
+
+1. It is located in the tree at position: ENTRY/definition/
+
+2. The "@" indicates that this is an attribute of the concept "definition".
+
+3. The name of the attribute is "version".
+
+4. Since it is "required", that means this attribute has to be added so that the resulting NeXus file is compliant with the NeXus definition "NXoptical\_spectroscopy".
+
+5. The "NX\_CHAR" indicates the datatype. This should be a string: "The preferred string representation is UTF-8" (more information see [here](<https://manual.nexusformat.org/nxdl-types.html>))
+
+![image.png](<./attachments/51dc82f9f0f5ec2f-image.png>)
+
+Now the python script has to be extended in the following:
+
+```
+f['/entry/definition'].attrs['version'] = 'v2024.02'
+```
+
+This h5py command adds the attribute named "version" with the value "v2024.02" to the HDF5 dataset called  "/entry/definition". The same is done for the URL attribute:
+
+```
+f['/entry/definition'].attrs['URL'] = 'https://github.com/FAIRmat-NFDI/nexus_definitions/blob/f75a29836431f35d68df6174e3868a0418523397/contributed_definitions/NXoptical_spectroscopy.nxdl.xml'
+```
+
+For your use case, you may want to use a different version of the NeXus definitions, since these are changed over time. In the following, it is shown where to obtain the correct version and URL.
+
+__Get the values: *version* and *URL*__
+
+At the time, you create the NeXus definition. Go to the page of the respectively used NeXus concept, i.e. [NXoptical_spectroscopy](<https://fairmat-nfdi.github.io/nexus_definitions/classes/contributed_definitions/NXoptical_spectroscopy.html>)
+
+Scroll down until you find "**NXDL Source**:" and follow this link, i.e. [NXoptical_spectroscopy.nxdl.xml](<https://github.com/FAIRmat-NFDI/nexus_definitions/blob/fairmat/contributed_definitions/NXoptical_spectroscopy.nxdl.xml>)
+
+This is the GitHub website, in which the latest (FAIRmat) NeXus definition of NXoptical\_spectroscopy is stored in the NeXus definition language file (.nxdl). The information is structured in the xml format.
+
+Now you have to copy the permalink of this file. Go to the top right side of the website. Find the Menu made by 3 dots:
+
+![image.png](<./attachments/c6ab2f4b925aed27-image.png>)
+
+Copy the permalink and insert it as value for the "URL" attribute (Step 1, Red box in the image)
+
+Go to "nexus\_definitions" (Step 2, Red box in the image)
+
+![image.png](<./attachments/d8e727b3b32dcbb9-image.png>)
+
+On the right side, you should see below "Releases" the "tags" (Red box in the image). Follow this link.
+
+Copy the latest tag, which should look similar to "v2024.02". Insert it as value for the "version" attribute.
+
+__Disclaimer__
+When specifying this version tag, it would be better to include the "GitHub commit id" as well. In this way, a [pynxtools generated version tag](https://github.com/FAIRmat-NFDI/pynxtools/blob/c13716915bf8f69068c3b94d1423681b580fd437/src/pynxtools/_build_wrapper.py#L17) might look like this:
+`v2022.07.post1.dev1278+g1d7000f4`. If you have pynxtools installed, you can get the tag by:
+
+```python
+>>> from pynxtools import get_nexus_version
+>>> get_nexus_version()
+'v2022.07.post1.dev1284+gf75a2983'
+```
+
+
+
+### Adding a datafield
+
+Two attributes were added to "ENTRY/definition", both of which were required. By now, this part of the NeXus file fulfills the requirements of the application definition NXoptical\_spectroscopy.
+
+The next required concept of [NXoptical_spectrsocopy](https://fairmat-nfdi.github.io/nexus_definitions/classes/contributed_definitions/NXoptical_spectroscopy.html) is "**experiment\_type"**.
+
+**experiment\_type**: (required) [NX\_CHAR](<https://fairmat-nfdi.github.io/nexus_definitions/nxdl-types.html#nx-char>)
+
+1. It is located in the tree at position: ENTRY/
+
+2. There is no "@" in front of "**experiment\_type"**. So, this may be a group or a datafield.
+
+3. The name of this group/datafield is "**experiment\_type**".
+
+4. The "required" indicates that this group/datafield has to be added to be in line with the NeXus definition "NXoptical\_spectroscopy".
+
+5. The "NX\_CHAR" indicates the datatype. This should be a string: "The preferred string representation is UTF-8" (more information see [here](<https://manual.nexusformat.org/nxdl-types.html>)).
+
+6. The "NX\_CHAR" indicates that this is a datafield. It is NOT a group.  
+    A group is a NeXus class. "NXentry" is for example a NeXus class, while "NX_CHAR" indicates the datatype of the field.
+    Whether or not the underscore "_" is present after NX, indicates therefore if it is a NeXus class or datafield.
+
+Read the documentation at "▶ Specify the type of the optical experiment. ..." by extending it via click on the triangle symbol. You should see something like this:
+
+![image.png](<./attachments/5cbd8c6a1ca227df-image.png>)
+
+There, the value of the datafield has to be one of the shown list, since it is an enumeration (e.g. "transmission spectroscopy"). Note that this is case sensitive.
+
+Therefore, the python script has to be extended by:
+
+```
+f['/entry/experiment_type'] = 'transmission spectroscopy'
+```
+
+
+
+
+
+### Adding a group
+
+The first required group in NXoptical\_spectroscopy on the "ENTRY/" level is "**INSTRUMENT**: (required) [NXinstrument](<https://fairmat-nfdi.github.io/nexus_definitions/classes/base_classes/NXinstrument.html#nxinstrument>) [⤆"](<https://fairmat-nfdi.github.io/nexus_definitions/classes/base_classes/NXentry.html#nxentry-instrument-group>)
+
+1. It is located in the tree at position: NXentry/
+
+2. There is no "@" in front of "**INSTRUMENT"** and because the "NXinstrument" is a NeXus class, this has to be implemented as group in the python script.
+
+3. The "required" indicates that this group has to be added to be in line with the NeXus definition "NXoptical\_spectroscopy".
+
+4. The "NXinstrument" indicates that it is a NeXus class (or group in python), as it starts with "NX" - without an underscore "_". It can also not be found at the [data types](https://manual.nexusformat.org/nxdl-types.html#data-types-allowed-in-nxdl-specifications).
+
+5. As this is a group, attributes or values may be assigned to it.
+
+6. As this is a group, it can contain many datafields or groups.
+
+7. The uppercase notation of "**INSTRUMENT**" means:
+
+    1. You can give INSTRUMENT [almost](https://manual.nexusformat.org/datarules.html) any name, such as "abc" or "Raman\_setup" (see "regex" or regular expression).
+
+    2. You can create as many groups with the class NXinstrument as you want. Their names have to be different.
+
+    3. For more information see the [NeXus rules](../learn/nexus-rules.md)
+
+The respective python code to implement a NXinstrument class (or equivalently in python group) with the name "experiment\_setup\_1" is:
+
+```
+f.create_group('/entry/experiment_setup_1')
+f['/entry/experiment_setup_1'].attrs['NX_class'] = 'NXinstrument'
+```
+
+The first line creates the group with the name "experiment\_setup\_1".
+
+The second line assigns this group the attribute with the name "NX\_class" and its value "NXinstrument".
+
+
+
+
+
+### Finishing the NeXus file
+
+This has to be done by using the respective NeXus definition website:
+
+[NXoptical_spectroscopy](<https://fairmat-nfdi.github.io/nexus_definitions/classes/contributed_definitions/NXoptical_spectroscopy.html>)
+
+And by searching for all "required" entries. The next required entries are located inside the NXinstrument class:
+
+1. **beam\_TYPE**: (required) [NXbeam](<https://fairmat-nfdi.github.io/nexus_definitions/classes/base_classes/NXbeam.html#nxbeam>) [⤆](<https://fairmat-nfdi.github.io/nexus_definitions/classes/base_classes/NXinstrument.html#nxinstrument-beam-group>)
+
+2. **detector\_TYPE**: (required) [NXdetector](<https://fairmat-nfdi.github.io/nexus_definitions/classes/base_classes/NXdetector.html#nxdetector>) [⤆](<https://fairmat-nfdi.github.io/nexus_definitions/classes/base_classes/NXinstrument.html#nxinstrument-detector-group>)
+
+Both are groups. "**beam\_TYPE"** could be named: "beam\_abc" or "beam\_Raman\_setup". Use the knowledge above to extend the python script to create those NeXus file entries.
+
+__Note for required NeXus concepts__
+
+Above in the definition of NXoptical\_spectroscopy, you as well may found a required entry "**depends\_on**: (required) [NX\_CHAR](<https://fairmat-nfdi.github.io/nexus_definitions/nxdl-types.html#nx-char>) [⤆"](<https://fairmat-nfdi.github.io/nexus_definitions/classes/contributed_definitions/NXcoordinate_system.html#nxcoordinate-system-depends-on-field>). This is at the level of "ENTRY/reference\_frames/beam\_ref\_frame". If you don't have the group "**beam\_ref\_frame"** because this is "optional", then you don't need to have this field.
+
+
+
+[_Continue by validating the NeXus file_](validate-nexus-file.md)
+
+## Feedback and contact
+
+1. Best way is to contact the FAIRmat team directly by creating a [Github Issue](https://github.com/FAIRmat-NFDI/nexus_definitions/issues/new).
+
+2. ron.hildebrandt(at)physik.hu-berlin.de
+
+
+

docs/how-tos/attachments/5cbd8c6a1ca227df-image.png

@@ -0,0 +1,80 @@
+
+This lists some notes for installation of nxvalidate on Ubuntu and Windows. For windows, the installation of the XML2 library was not sucessful. This should be possible, but could not reproduced yet.
+
+
+
+# cnxvalidate installation on Ubuntu 22.04
+
+These commands install nxvaldiate on a fresh Ubuntu 22.04 system (tested with Linux running from USB stick).
+
+```
+sudo apt-get update
+sudo apt-get install git
+sudo apt-get install build-essential
+sudo add-apt-repository universe
+sudo apt-get install libhdf5-serial-dev
+sudo apt-get -y install pkg-config
+sudo apt upgrade -y
+sudo apt-get -y install cmake
+sudo apt-get install libxml2-dev
+
+mkdir nexusvalidate
+cd nexusvalidate
+git clone https://github.com/nexusformat/cnxvalidate.git
+cd cnxvalidate/
+mkdir build
+cd build/
+cmake ../
+make
+```
+
+# cnxvalidate installation on windows:
+
+## -- CMAKE
+
+[https://cmake.org/download/](<https://cmake.org/download/>)
+
+\--> [cmake-3.30.2-windows-x86\_64.msi](<https://github.com/Kitware/CMake/releases/download/v3.30.2/cmake-3.30.2-windows-x86_64.msi>)
+
+Install with .msi
+
+## -- HDF5
+
+Download **hdf5-1.14.4-2-win-vs2022\_**[**cl.zip**](<http://cl.zip>)** from **[https://www.hdfgroup.org/downloads/hdf5/](<https://www.hdfgroup.org/downloads/hdf5/>)
+
+unzip the .zip file
+
+put the file into the folder
+
+```
+C:\hdf5
+```
+
+(can be named differently, but no spaces are allowed for this path)
+
+```
+set PATH=%PATH%;C:\your\path\here\
+```
+
+## -- libiconv
+
+[https://github.com/vovythevov/libiconv-cmake](<https://github.com/vovythevov/libiconv-cmake>)
+
+```
+git clone
+```
+
+cd to downloaded directory
+
+```
+mkdir build
+cd build
+cmake ..
+```
+
+## -- XML2
+
+??? Unsolved...
+
+Please create GitHub issue [here](https://github.com/FAIRmat-NFDI/pynxtools/issues/new) if you could solve this.
+