HDF5 versus (ome-)zarr

[ctrueden]

Have you considered adopting/extending ome-zarr?

https://doi.org/10.1007/s00418-023-02209-1

The broader microscopy community is pushing to standardize on zarr rather than HDF5 because HDF5 files have performance bottlenecks with multithreaded writing. Whereas with zarr, particularly zarr v3 IIUC, data can be written to multiple files distributed in a way balanced decently for both read and write performance.

See also:

• https://imagesc.zulipchat.com/#narrow/channel/328251-NGFF/topic/NGFF.20to.20store.20data.20from.20Brillouin.20microscopy

CC @joshmoore

Edit: Also, I forgot to mention: while Fiji has plugins to read arbitrary HDF5 data, they are not currently well maintained. Whereas there is a lot of effort currently toward making Fiji's support for (ome-)zarr better.

[jkh1]

Also zarr has support in many programming languages either directly (see https://zarr.dev/implementations/) or indirectly through wrapping of libraries that include it (e.g. NetCDF, GDAL)

[bevilacqc]

@ctrueden @jkh1 Thank you for your comments! We are actually looking into moving to Zarr (it seems it can support the structure we need).
We ideally want to have a single file, so we are looking with @SebastianHambura at the Zip storage. Do you have any experience with it or are aware of any drawback?

[jkh1]

@bevilacqc One potential drawback of zip storage is that compression would be useless in some cases and could even result in larger files due to poor compression of already compressed data. For such cases, compression would have to be turned off. Others seem already discussed under the link you gave.