Improvements to the NeXus parsing/normalization in NOMAD

[lukaspie]

There was a discussion in the TF meeting on how to restructure the representation of NeXus data in NOMAD. Discussion points:

• Building a vocabulary of methods for the different NeXus application definitions to populate the entry_type (see screenshot). For example, NXem should be mapped to something like Electron microscopy (EM). Is connected to the ontology service developed in Ontology Service Integration #680.

• Should each NXentry be parsed into one NOMAD Entry? There was a favourable vote on this in the TF meeting. Only question is how to keep links between the different entries from one NeXus file.

• Identifiers: should a new NXsample entry be created if a sample is found? This could then be linked through the same mechanism that we use right now (where we just create an abstract reference).

• Still open: how to deal with different versions of the NeXus schema? For example, if a user uploads a file which contains non-existent NeXus base classes (which we were working on before, but decided to drop eventually in the standardization). This will hopefully not be such a huge problem going forward as now the standardization is actually done (still needs the NeXus release though).

Feel free to add @FAIRmat-NFDI/areab @Pepe-Marquez.

[rettigl]

• Should each NXentry be parsed into one NOMAD Entry? There was a favourable vote on this in the TF meeting. Only question is how to keep links between the different entries from one NeXus file.

Maybe we anyways need to think about the referencing mechanism once more. Right now, the "ID" of a NeXus entry remains empty, and if an "identifier_entry" exists in Nexus, it create an (empty) reference entry. Rather, I would use information from the Nexus entry such as an identifier_entry to fill the ID field, and maybe also some other fields to fill "description" etc.

• Identifiers: should a new NXsample entry be created if a sample is found? This could then be linked through the same mechanism that we use right now (where we just create an abstract reference).

In the context of revising the referencing mechanism, this is also something we should think about. If we find an existing and linkable sample entry, how do we deal with this information? Should we add it to the Nomad entry in case it is missing in the Nexus file (e.g. atom types etc.)? What happens if the data in the sample entry change then? Could we make this dynamic, i.e. dynamically link fields in one Nomad Entry to fields in another?

• Still open: how to deal with different versions of the NeXus schema? For example, if a user uploads a file which contains non-existent NeXus base classes (which we were working on before, but decided to drop eventually in the standardization). This will hopefully not be such a huge problem going forward as now the standardization is actually done (still needs the NeXus release though).

I believe this to be less of an issue. In case the file does not fit the installed definitions, those parts should ideally just fail parsing and be missing from the Nomad entry.

[lukaspie]

• Still open: how to deal with different versions of the NeXus schema? For example, if a user uploads a file which contains non-existent NeXus base classes (which we were working on before, but decided to drop eventually in the standardization). This will hopefully not be such a huge problem going forward as now the standardization is actually done (still needs the NeXus release though).

I believe this to be less of an issue. In case the file does not fit the installed definitions, those parts should ideally just fail parsing and be missing from the Nomad entry.

The bigger issue is with reprocessing, where a file may have been right at some point, but then we change the NeXus/pynxtools version in a deployment and then it will fail on reprocessing. This is however a general NOMAD issue and there are already conversation how to address this (e.g., by attaching the schema to the data instances).

[TLCFEM]

Still open: how to deal with different versions of the NeXus schema?

We are gradually implementing a mechanism to store versioned definition packages in mongodb, so even if the definition changes in future, the data then processed can still be fetched and interpreted.