Tame the eventual memory hungryness of the dataconverter and NOMAD parser for large datasets

[mkuehbach]

• Observed for 15.5 GiB NeXus/HDF5 NXem dataset, essentially all content in one 3d array which uncompressed is 32GiB, the array if stored chunked, upon parsing multiple times the uncompressed payload is allocated causing the parsing to break on small systems.

Suggestions, we should check for inefficiencies, assuming what happens when a dataset is large:

• Compute summary statistics and finiteness via taking advantage of chunked layout, chunk-by-chunk
• Could be useful to drop processing statistics for contiguously stored datasets if larger than fraction of maximum allocated memory accessible to the user in the deployment, like "drop of load"
• Access h5 objects metadata directly rather than via np.shape() calls

[mkuehbach]

import numpy as np
import h5py

with h5py.File("data.h5", "r") as f:
    dset = f["dataset_name"]
    
    min_val = np.inf
    for chunk_sel in dset.iter_chunks():
        chunk = dset[chunk_sel]
        # should decompress automatically e.g., when using build-in h5 compression filters like deflate   
        min_val = min(min_val, chunk.min())

Suggests also to have pynxtools-plugin writing arrays above some threshold always chunked instead of contiguous.

Indeed, https://github.com/FAIRmat-NFDI/pynxtools/blob/f41e66a43d385cd619d3f1e08c855beeb0b17a91/src/pynxtools/nomad/parser.py#L138C5-L138C30 these load large datasets fully into memory.

[mkuehbach]

• Should add guidance to users via docs, like "Tips for working with large datasets"