Storage Benchmarks

[kmpaul]

There is already the https://github.com/pangeo-data/storage-benchmarks repository, which we can build on (possibly move into this repo). I think that these benchmarks should consider different formats:

• zarr
• hdf5
• netcdf

And I think we need to compare these to their "idealized" use cases, which are independent I/O (i.e., each process reads/writes from/to its own file) for zarr and MPI-IO (each process reads/writes from/to the same file) for hdf5 and netcdf.

These benchmarks should be run on different platforms and storage systems (HPC with GPFS or Lustre, AWS S3, GCS, etc.).

What all do we need for this?

[rabernat]

This repo has a lot of ideas to start from: https://github.com/rabernat/zarr_hdf_benchmarks

@andersy005 has been running it on Cheyenne

[kmpaul]

@rabernat Thanks! I think this is very similar to the kinds of benchmarks that IOR conducts with C-based code. I have tried doing something similar with the Python code in the past, and I have not seen scaling. However, I have seen scaling with the C libraries. I don't know why this is, yet.

[kmpaul]

Haiying Xu (@halehawk) now has a fork of IOR in the NCAR organization (NCAR/ior). I am going to have IOR scripts added to this repo and I/O results on Cheyenne using IOR/Z5.

[kmpaul]

I believe some of this has been address by #44.