You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/how-tos/validate-nexus-file.md
+36-47Lines changed: 36 additions & 47 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,6 @@
1
-
# Validate NeXus files
1
+
# Validating NeXus files
2
+
3
+
Note: This is a how-to guide for using different tools to validate NeXus files. If you want to learn more about how validation is done in `pynxtools`, please visit the [explanation page](../learn/nexus-validation.md).
2
4
3
5
# The goal
4
6
@@ -9,25 +11,21 @@ Use a tool to validate NeXus files to a given set of NeXus definitions:
9
11
2.[NIAC](https://manual.nexusformat.org/)
10
12
11
13
12
-
13
14
# Validation of a .nxs file
14
15
15
-
The validity of NeXus file is fundamental, to ensure FAIR data. Without specific requirements, it is not possible to understand the data. What type of experiment? What Laser Wavelength? Which voltage? What data is represented at all in the table? What is the unit of the value? Which ISO norm does this refer to? Where was this measured? Which year was this measured?
16
+
The validity of NeXus files is fundamental to ensure FAIR data. Without specific requirements, it is not possible to understand the data. What type of experiment? What Laser Wavelength? Which voltage? What data is represented at all in the table? What is the unit of the value? Which ISO norm does this refer to? Where was this measured? Which year was this measured?
16
17
17
18
Therefore you have enter all required fields in the NeXus definition. The requirements are set by the community via workshops or at conferences. You can as well comment the NeXus definitions, to initiate or propose changes/additions. Go to the NeXus definition, and sign-up/log-in and give us some feedback (Red boxes in the image. Expand this panel on the left by clicking on the arrow symbol).
ftentimes, there will be errors in a generated NeXus file (be it by hand or automatically): Typos, missing required concepts, missing attributes, using the incorrect datatype or format (e.g., array instad of list, float instead of integer, etc.). Therefore, a validation is required, to ensure that the data you want to share, is FAIR.
21
+
Oftentimes, there will be errors in a generated NeXus file (be it by hand or automatically): Typos, missing required concepts, missing attributes, using the incorrect datatype or format (e.g., array instad of list, float instead of integer, etc.). Therefore, a validation is required, to ensure that the data you want to share, is FAIR.
21
22
22
-
The NeXus file is valid, if it complies with the respective NeXus application definition.
23
+
The NeXus file is valid if it complies with the respective NeXus application definition.
23
24
24
25
This validation is done by software.
25
26
26
27
27
-
28
-
29
-
30
-
# 1. Validation software
28
+
# Validation software
31
29
32
30
There are right now three tools, which can be used for validation of NeXus files. All are different and have individual advantages or disadvantages:
33
31
@@ -37,41 +35,34 @@ There are right now three tools, which can be used for validation of NeXus files
37
35
38
36
3. punx
39
37
40
-
41
38
Open software is usually shared on Github - There you find usually the most accurate information, as documentation sometimes lags behind. There you see a box with folders and files. Below is the content of the README.md file displayed. This usually shows instructions for installation and handling of the software.
42
39
43
40
Here are the GitHub links for the thee software packages:
WARNING: Field /entry/instrument/beam_incident/wavelength/@units written without documentation.
108
99
...
109
100
```
110
101
111
-
# 2.1.1 Installation verfiy_nexus
102
+
# 1.1.1 Installation of verify_nexus
112
103
113
-
Aside of read_nexus, there is a second functionality from pynxtools. This method is currently in development (Aug 2024). This function is called: verify\_nexus. Therefore, you have to use the development install, until this function is published.
104
+
Aside from read_nexus, there is a second functionality from pynxtools. This method is currently in development (Aug 2024). This function is called: verify\_nexus. Therefore, you have to use the development install, until this function is published.
114
105
115
106
Do this to install pynxtools with verify\_nexus
116
107
@@ -143,7 +134,7 @@ Options:
143
134
```
144
135
145
136
146
-
# 2.1.2 Using verify\_nexus
137
+
# 1.1.2 Using verify_nexus
147
138
148
139
Open your terminal. Assuming there is a folder at:
149
140
@@ -159,9 +150,9 @@ For Windows:
159
150
C:\nexusvalidation
160
151
```
161
152
162
-
Put into this folder your NeXus file, for example the[Raman.nxs file](https://zenodo.org/records/13373909/files/Raman.nxs?download=1).
153
+
Put into this folder your NeXus file, for example this[Raman.nxs file](https://zenodo.org/records/13373909/files/Raman.nxs?download=1).
163
154
164
-
Use verify nexus with the command:
155
+
Use verify_nexus with the command:
165
156
166
157
```
167
158
verify_nexus C:\nexusvalidation\Raman.nxs
@@ -179,9 +170,7 @@ Invalid: The entry `entry` in file `Raman.nxs` is NOT a valid file according to
179
170
```
180
171
181
172
182
-
183
-
184
-
# 2.2 read_nexus
173
+
# 1.2 read_nexus
185
174
186
175
The command used is:
187
176
@@ -230,7 +219,7 @@ The second example was for the "software\_TYPE" attribute @URL entry in the "NXo
230
219
231
220
232
221
233
-
# 2.2.1 Installation read_nexus
222
+
# 1.2.1 Installation of read_nexus
234
223
235
224
This is installed with pip:
236
225
@@ -239,7 +228,7 @@ pip install pynxtools
239
228
```
240
229
241
230
242
-
# 2.2.2 Using the read\_nexus function
231
+
# 1.2.2 Using the read_nexus function
243
232
244
233
Open your terminal. Assuming there is a folder at:
245
234
@@ -288,7 +277,7 @@ For Axis #0, 1 axes have been identified: [<HDF5 dataset "spectrum_data_x_Raman"
288
277
DEBUG: For Axis #0, 1 axes have been identified: [<HDF5 dataset "spectrum_data_x_Raman": shape (1600,), type "<f8">]
289
278
```
290
279
291
-
Search for filed which are not found in the NeXus definiton by searching for the line: "DEBUG: NOT IN SCHEMA". Recheck the used NeXus definition to eliminate the problem. Be careful with upper and lower case notation and correct spelling.
280
+
Search for fields which are not found in the NeXus definition by searching for the line: "DEBUG: NOT IN SCHEMA". Recheck the used NeXus definition to eliminate the problem. Be careful with upper and lower case notation and correct spelling.
292
281
293
282
Keep in mind, that the output provides quite some information. This is useful for software development, but may be a bit too much for validation purposes.
294
283
@@ -303,7 +292,7 @@ Similar features as the tables and messages provided from punx and cnxvalidate f
303
292
304
293
305
294
306
-
# 3. cnxvalidate
295
+
# 2. cnxvalidate
307
296
308
297
This package is written in C. It is allows a command line evocation like:
and indicates the entry of the .nxs file, which is incorrect and what the respective problem is. It also points to the NeXus definition (.nxdl.xml file), in which this conflict was found.
327
316
328
-
# 3.1. Installation (Linux only)
317
+
# 2.1. Installation (Linux only)
329
318
330
319
This [did not work for me on windows](/installation_notes_nxvalidate.md) (The problem was the software cmake couldn't fine the libxml2 library. Though, if you solve this, this maybe work on windows).
331
320
@@ -335,7 +324,7 @@ The installation process has to be build from source. This is eased significantl
Now you can start to validate your created [NeXus file](https://zenodo.org/records/13373909). But before the validation, we need to get a set of NeXus definitions, which we want to use as reference. This is done again by using git:
431
420
@@ -510,7 +499,7 @@ Now adjust the file creation, and add the respective fields to make your NeXus f
510
499
511
500
512
501
513
-
# 4. Punx - Python Utilities for NeXus HDF5 files
502
+
# 3. Punx - Python Utilities for NeXus HDF5 files
514
503
515
504
This is python package, and can therefore be used on Linux and Windows systems.
516
505
@@ -535,7 +524,7 @@ This is done by replacing REPORT with ={COMMENT,ERROR,NOTE,OK,TODO,UNUSED,WARN}
The output tables "findings" and "summary statistics" can be used to find error present in the NeXus file.
684
673
685
-
# 4.3. Example
674
+
# 3.3. Example
686
675
687
676
### Which NeXus definition?
688
677
@@ -756,7 +745,7 @@ The last error message:
756
745
757
746
can be ignored and is a bug right now. If this is the only Error message, then your NeXus file is compliant with the NeXus definitions and you can share and publish your data.
758
747
759
-
# 4.4. Further Notes
748
+
# 3.4. Further Notes
760
749
761
750
1. Punx only uses the NeXus definiton from the NIAC [NeXus definiton from the NIAC](<https://manual.nexusformat.org/>). The use of the [FAIRmat NeXus definition](<https://fairmat-nfdi.github.io/nexus_definitions/index.html#>) is not possible right now.
762
751
@@ -781,11 +770,11 @@ This tutorial showed:
781
770
782
771
### Recommended methods:
783
772
784
-
As pynxtools verify_nexus method is right now in development, [not all situations are covered right now](/testing-validation-tools.md). Therefore, the most reliable method right now is a combination of _Human Manual Validation_ + _1-2 Software solutions_.
773
+
As pynxtools verify_nexus method is right now in development, [not all situations are covered right now](/testing-validation-tools.md). Therefore, the most reliable method right now is a combination of _Human Manual Validation_ + _Software solutions_.
785
774
786
775
### Pynxtools Parsers:
787
776
788
-
For a specifically structured set of data, a parser can be written, which uses the meta data and a pre-structured meta data file, to create a NeXus file. Tough, the parser depends on: Experimental Technique and Setup and has therefore to be written invidiually. This is another functionallity of [pynxtools with plugins for the techniques](https://fairmat-nfdi.github.io/pynxtools/reference/plugins.html):
777
+
For a specifically structured set of data, a parser can be written, which uses the meta data and a pre-structured meta data file, to create a NeXus file. Tough, the parser depends on: Experimental Technique and Setup and has therefore to be written individually. This is another functionallity of [pynxtools with plugins for the techniques](https://fairmat-nfdi.github.io/pynxtools/reference/plugins.html):
0 commit comments