In order to support larger-than-memory embeddings, we need some tooling for performant lazy loading. A few features/characteristics:
Libraries for IO:
- TensorStore. this seems like it's the most flexible backend-wise
- xarray
- dask (@kephale noted some performace limitations with large numbers of chunks in the napari tiled rendering work)