NeXus compatibility

[ramav87]

We have been exploring Nexus compatibility. The main issue appears to be:

• original metadata should be a part of an NXCollection (or an NXobject if this does not parse correctly with the current NxEM parser for Nomad)
• the dimension list should not use h5 object references and should instead list the dataset or group name directly. Use of HDF references causes problems that lead to incompatibility with the current NeXus parser implementations.

[ramav87]

Here is what we need

1. pyNSID shoudl only write to a Nexus compatible file format
2. When we read the file back, we shoudl read it back as a sidpy dataset format
3. (future) perhaps we can think about a converter from sidpy to nexus data format as well

Here is sample code to write files in a way that is minimally Nexus compatible, assuming we are writing sidpy datasets of images (e.g., from ibw files that are read from SciFiReaders).

with h5py.File(h5_name, 'w') as h5_f:
    h5_group = h5_f.create_group('Measurement_Nexus')
    my_str = "NXsidpy"
    def_dset = h5_group.create_dataset('definition', data=my_str)
    
    for key in list(data.keys()):
        h5_g = h5_group.create_group(key)
        nsid.hdf_io.write_nsid_dataset(data[key], h5_g)
        #TODO: we need to update the nsid writer so it natively will make nxs compatible files.
        #But this is a temporary workaround for now.

        h5_g.attrs["NX_class"] = "NXdata"
        h5_g['generic'].copy("x", "axis_i")
        h5_g['generic'].copy("y", "axis_j")
        h5_g['generic']['generic'].attrs["NX_class"] = "NXdata"
        h5_g['generic'].attrs["signal"] = "generic"
        axes_refs = np.array(
            ['axis_j', 'axis_i'],
            dtype='object'
        )
        h5_g['generic'].attrs["axes"] = axes_refs
        h5_g['generic'].attrs["axis_i_indices"]=0
        h5_g['generic'].attrs["axis_j_indices"]=1
        h5_g['generic']['axis_i'].attrs["units"] = "nm"
        h5_g['generic']['axis_i'].attrs["long_name"] = "Width"

        h5_g['generic']['axis_j'].attrs["units"] = "nm"
        h5_g['generic']['axis_j'].attrs["long_name"] = "Length"

        h5_g.attrs["default"] = "generic"
        h5_f.attrs["default"] = key
        #h5_g['metadata'].attrs["NX_class"] = "NXobject"

        del h5_f['Measurement_Nexus'][key]['generic']['generic'].attrs["DIMENSION_LIST"] #delete dimension list attribute

    
    h5_group2 = h5_f.create_group('Measurement_sidpy')
    for key in list(data.keys()):
        h5_g2 = h5_group2.create_group(key)
        nsid.hdf_io.write_nsid_dataset(data[key], h5_g2)


print("---Converted ibw File to h5 Successfully---")

Then, the question is about reading it back as a sidpy dataset. Right now there are some checks that will fail that will need to be skipped; beyond that, we can recover the sidpy datasets like so (Load one of the above converted h5 files)

nsid_reader = sr.NSIDReader(h5_file) #read with nsid reader
data_nsid = nsid_reader.read()

data_c0 = data_nsid['Channel_000']

data_1_x = nsid_reader._h5_file['Measurement_Nexus/Channel_000']['generic']['x']
x_quantity = data_1_x.attrs["quantity"]
x_name = data_1_x.attrs["name"]
x_units = data_1_x.attrs["units"]
x_dim_type = data_1_x.attrs["dimension_type"]

data_1_y = nsid_reader._h5_file['Measurement_Nexus/Channel_000']['generic']['y']
y_quantity = data_1_y.attrs["quantity"]
y_name = data_1_y.attrs["name"]
y_units = data_1_y.attrs["units"]
y_dim_type = data_1_y.attrs["dimension_type"]

data_generic = nsid_reader._h5_file['Measurement_Nexus/Channel_000']['generic']['generic']
data_type = data_generic.attrs["data_type"]
data_quantity = data_generic.attrs["quantity"]
data_units = data_generic.attrs["units"]

#with this info we can reconstruct the sidpy dataset
data_c0.set_dimension(0, sid.Dimension(data_1_x[:],
                                          name=x_name, units=x_units, quantity=x_quantity,
                                          dimension_type=x_dim_type))

data_c0.set_dimension(1, sid.Dimension(data_1_y[:],
                                          name=y_name, units=y_units, quantity=y_quantity,
                                          dimension_type=y_dim_type))

data_c0.data_type = data_type
data_c0.quantity = data_quantity
data_c0.units = data_units

Of course, we need to test this for different types of datasets, and confirm full functionality, then update pyNSID, ScFiReaders and probably sidpy with these changes.