Split of `nexus` package into smaller packages

[TLCFEM]

#574

As mentioned in another thread, in order to achieve the goal that then processed data can still be interpreted even if the corresponding definition has evolved over time, nomad is implementing a mechanism to store used package definitions in mongodb as snapshots.

This mechanism cannot bypass the limitation that each mongodb document cannot exceed 16MB in size.
The nexus package is too big to be stored as a single package.

To address this issue, it is probably necessary to consider splitting the current nexus package into several smaller ones.
In principle, it should be feasible to parse a subset of xml files and collect the result into a single package.

Note on lazy loading: in general it is preferred that all built-in packages shall be loaded when the process starts, because the startup procedure needs to dump all packages to mongodb.
If anything is lazily loaded, it is probably necessary to manually check-in those into mongodb later on.

[mkuehbach]

Modularization of NeXus is an ongoing topic and currently mainly driven by FAIRmat.
As for the nexus_definitions in the FAIRmat-NFDI repo we are essentially free to group the current definitions on top of the current default --- appdefs, base classes, and contributed definitions, all assumed required and thus loaded --- into the schema to enable some grouping.

The benefit of having all definitions together is that arbitrary combinations of domain-specific work can be dealt with. For sure, if you have an installation where one knows that only a subset of definitions is required, e.g. SPM, one should ideally bundle this and nothing more but the more multi-method-supportive the oasis is the closer you arrive again at demanding almost all definitions.

A possible route for compression should be mentioned although it will only replace caching with making API calls to a PID resolving service:
Currently the NeXus schema is instantiated from reading XML files. However, once the NeXusOntologyV2 will have been published after the NeXus milestone release NXDL2025, there will be a PID for each NeXus concept. That could be used for instantiating the NeXus schema as well, as the PID resolves the concept and with it its docstring and properties like NeXus Data type etc.

[mkuehbach]

Update 2025/10/07:

• 43MB generate Metainfo instance data
• decision on the implementation still in debated on the nomad side, potentially, each section an own document in the mongodb

[TLCFEM]

decision on the implementation still in debated on the nomad side, potentially, each section an own document in the mongodb

Although technically feasible, performance improves when related sections are grouped into reasonably sized packages, as the definition is stored per package base.