Lazy loading speed of NASA Earthdata with kerchunk vs virtualizarr reference files

[DeanHenze]

Hi all,

Dean here working out of PO.DAAC. We currently host several virtual reference files for some of our data sets made with kerchunk, and are excited to start making the reference files with virtualizarr instead. Love the easy syntax! I did want to bring something to your attention though and learn your thoughts.

Before the code, long story short is that I created two reference files for our OSTIA-UKMO-L4-GLOB-REP-v2.0 data set (global gridded data, ~15,000 netCDF files, ~250 GB on disk, ~11 TB uncompressed), one using kerchunk and the other with virtualizarr. Lazy loading the data with the kerchunk file takes a few seconds, while using the virtualizarr file takes 4-6 minutes. Some how, getting a subset of the data and plotting takes the same amount of time with either reference file. Still though, seems like it should be totally doable to lazy load the data in a shorter amount of time if kerchunk can do it?

As a minimal working example I cannot create the ref files for the full time series here, so I opted for 400 files (ends up taking about 5 minutes to make the files each with virtualizarr and kerchunk. No parallel computing as I'm trying to stick to a minimal working example). Also I use the earthaccess package to manage NASA credentials, find S3 endpoints, etc. So hopefully someone on the virtualizarr team has an earthaccess account?

Here is the minimal working environment

##----------------
## Minimal install (using Python 3.12)
##----------------
# kerchunk==0.2.7
# xarray==2025.1.2
# h5py==3.13.0
# ujson==5.10.0
# earthaccess==0.14.0
# matplotlib==3.10.0
# virtualizarr==1.2.0

import os
import ujson

import fsspec
import earthaccess

import xarray as xr
from kerchunk.hdf import SingleHdf5ToZarr
from kerchunk.combine import MultiZarrToZarr
from virtualizarr import open_virtual_dataset

import matplotlib.pyplot as plt

Next, use earthaccess to access data set metadata on PO.DAAC

# Get Earthdata creds
earthaccess.login()

# Get AWS creds
fs_data = earthaccess.get_s3_filesystem(daac="PODAAC")

# Locate file information / metadata:
granule_info = earthaccess.search_data(
    short_name="OSTIA-UKMO-L4-GLOB-REP-v2.0",
    count=400
    )

s3_endpoints = [g.data_links(access="direct")[0] for g in granule_info]

fobjs = earthaccess.open(granule_info) # Gets file-like objects for use with kerchunk

Create ref file with kerchunk

# Individual references:
single_refs_kerchunk = [
    SingleHdf5ToZarr(fobj, ep, inline_threshold=0).translate() 
    for fobj, ep in zip(fobjs, s3_endpoints)
    ]

# Combined reference:
kwargs_mzz = {'remote_protocol':"s3", 'remote_options':fs_data.storage_options, 'concat_dims':["time"]}
combined_ref_kerchunk = MultiZarrToZarr(single_refs_kerchunk, **kwargs_mzz).translate()

# Save reference to JSON:
with open("ref_combined_kerchunk.json", 'wb') as outf:
    outf.write(ujson.dumps(combined_ref_kerchunk).encode())

Create ref file with virtualizarr

# Create single ref files:
singe_refs_virtualizarr = [
    open_virtual_dataset(ep, indexes={}, reader_options={"storage_options": fs_data.storage_options})
    for ep in s3_endpoints
    ]

# Create combined ref file and save to json:
combined_ref_virtualizarr = xr.combine_nested(singe_refs_virtualizarr, concat_dim='time', coords='minimal', compat='override', combine_attrs='drop_conflicts')
combined_ref_virtualizarr.virtualize.to_kerchunk("ref_combined_virtualizarr.json", format='json')
##-----------------------------------------------------------------------------

Last, open the data using each reference file

def opendf_withref(ref, fs_data):
    """
    Wrapper function to open data with xarray, using a ref file. "ref" is a reference file or object. 
    "fs_data" is a filesystem with credentials to access the data files. 
    """
    storage_opts = {"fo": ref, "remote_protocol": "s3", "remote_options": fs_data.storage_options}
    m = fsspec.filesystem('reference', **storage_opts).get_mapper('')
    return xr.open_dataset(
        m, engine="zarr", chunks={},
        backend_kwargs={"consolidated": False}
        )

data_kerchunk = opendf_withref("ref_combined_kerchunk.json", fs_data)
print(data_kerchunk)

data_virtualizarr = opendf_withref("ref_combined_virtualizarr.json", fs_data)
print(data_virtualizarr)

I don't have the code that timed the lazy loading in this post, but you can see the results in my Jupyter notebook here (using %%time):
https://github.com/DeanHenze/cloud_earthdata/blob/main/virtualizarr/kerchunk_vs_virtualizarr.ipynb

It takes a few hundred milliseconds for the kerchunk file, vs ~5 seconds for the virtualizarr file.

I did my best to create a minimal working example, but now that I'm typing it all out I can see it's quite long. Still, I hope this can stimulate some discussion. Also, I'm trying to figure out if this is even a relevant issue currently, e.g. will saving things in icechunk format solve this?

Thanks,

• Dean

[TomNicholas]

Thanks for raising this @DeanHenze - this should be fixable.

Lazy loading the data with the kerchunk file takes a few seconds, while using the virtualizarr file takes 4-6 minutes.

Weird. It's important to be clear that there is no such thing as a "virtualizarr file" - you are comparing using the virtualizarr package vs using the kerchunk package, but in both cases you are using them to create a kerchunk-format json references file.

This means at the point you open the references file, virtualizarr is no longer involved. Therefore the only possible way you can be seeing a speed difference is if the contents of the reference file created by virtualizarr and the reference file created by kerchunk are not identical.

This seems plausible - I imagine if you examine the contents of your two json files carefully you will see some kind of difference in chunking or in which variables are inlined. There are performance footguns here (some of which I've recently been working on removing in #477), and I have some guesses as to what you might find.

Some how, getting a subset of the data and plotting takes the same amount of time with either reference file.

Interesting - a clue.

So hopefully someone on the virtualizarr team has an earthaccess account?

I don't have one, but if you can create the two kerchunk-formatted references files we should be able to iterate from there without me needing an earthaccess account.

---

will saving things in icechunk format solve this?

In theory loading the same data referenced from icechunk can be a lot faster than loading it from a kerchunk reference file. It also should have better scaling properties for large numbers of references. But there will be some similar performance footguns to avoid. It would therefore be worth trying this same experiment with icechunk too.

[TomNicholas]

For the record here's my prediction as to what you will find - don't read this until after you have a look at the files if you don't want my guesses to cloud your judgement!

My prediction

• You will find that the data for certain variables are inlined in the kerchunk-created json file which are not inlined in the virtualizarr-created json file.
• Specifically kerchunk will have inlined every variable smaller than 500 bytes (the default), but virtualizarr won't have inlined any variables, because you didn't specify any loadable_variables.
• This means that when opening the virtualizarr-created references file many more variables have to be requested from object storage than when opening the kerchunk-created references file.
• You should be able to create two identical files by specifying to loadable_variables the same set of variables which kerchunk chose to inline. (It will probably be all the scalar and 1-dimensional variables.)
• Once you have created an identical json file, reading data from it should take an identical amount of time.
• If instead you write all the references (and loadable variables) to icechunk, loading the data should be even faster.
• It's possible that even if you don't explicitly use loadable_variables, icechunk will still inline the small variables for you, and be fast in either case.

If I'm right this could potentially form the basis of an FAQ question.

[DeanHenze]

Hi Tom,

Thanks for getting back to me so quickly and writing out your thoughts in detail.

It's important to be clear that there is no such thing as a "virtualizarr file" - you are comparing using the virtualizarr package vs using the kerchunk package, but in both cases you are using them to create a kerchunk-format json references file.

Yes understood, just saying "kerchunk file" vs "virtualizarr file" for convenience.

Looks like using the loadable_variables arg and passing the three coordinate variables (all 1D) solved it, the full record now opens in a few seconds. Wow it's almost unbelievable what a difference those three little vars are making.

I think this could save others a lot of time if added as an FAQ. For my technical level, I wouldn't have been able to look at the API and determine what effect the loadable_variables arg would have. Not that you are trying to cater to my level, just sharing some thoughts. If there are other tricks like this, maybe the FAQ question could be expanded to "tips and tricks for the lay person to optimize their reference file".

Either way, thank you @TomNicholas I'm excited to start implementing our emerging workflows at PO.DAAC using virtualizarr, and ultimately trying out Icechunk.

[TomNicholas]

Looks like using the loadable_variables arg and passing the three coordinate variables (all 1D) solved it, the full record now opens in a few seconds.

Great!

I think this could save others a lot of time if added as an FAQ. For my technical level, I wouldn't have been able to look at the API and determine what effect the loadable_variables arg would have.

The next release of VirtualiZarr (currently on the develop branch) should actually solve this for you, as the deafult behaviour was changed (by #477).

I'm excited to start implementing our emerging workflows at PO.DAAC using virtualizarr, and ultimately trying out Icechunk.

Excellent, please don't hesitate to raise more issues if you run into more problems.