Supporting VirtualiZarr 2.x

[betolink]

We need to refactor our dmrpp<->virtualizarr code to support VirtualiZarr 2.x, this means removing some of the code earthacess implemented to parallelize the dmrpp concatenation since that is now directly supported by virtualizarr.

tasks

• Replace fsspec with Obstore as i/o driver, we should import S3Store and HTTPStore
  • If a user is requesting direct access we need to:
  • Inspect the first granule looking for the collection_concept_id, search for the parent collection S3Credentials endpoint.
  • Create an S3Store instance with the S3credentials
  • If a user requests out of region access:
    • Configure an HTTPStore with the bearer token
• Call VirtualiZarr open_virtual_mfdataset using the Obstore store, parallel="Dask"

Question: for S3 Obstore's S3Store uses the bucket prefix and shares that for a list of S3 URLs , will that also work with
HTTPStore?

@maxrjones wrote a nice notebook with all these steps that would be a good starting point: https://gist.github.com/maxrjones/2a3db891f184a16289e2cb8c2961f792

cc @owenlittlejohns @chuckwondo @danielfromearth

[danielfromearth]

Built further upon the notebook from @maxrjones with additional description and visualization at the end, for a fully demonstration:
https://nasa.github.io/ASDC_Data_and_User_Services/PREFIRE/how_to_generate_and_read_virtual_datacube-for-PREFIRE.html

[owenlittlejohns]

@TomNicholas - We've got a mostly working version of this working with obstore, but had a quick question: do we still need the load option to materialise the indices etc? For reference, here's how it's used, and here's the documentation for it. @betolink and I reckon we can just cut that entire condition, due to the updates to virtualizarr in 2.x, but wanted to confirm. Thanks!

[danielfromearth]

Yeah, I'd agree with separating functionality for those two use-cases.

The one case that seems like a blending of the two (creating a virtual cube and loading) is the DMR++ parsing case:
The way we are reading a data collection that doesn't have a pre-generated virtual datacube but has DMR++ for each data file, involves creating and loading a virtual dataset for each file on the fly (it's what's shown in the current earthaccess-virtual dataset tutorial notebook). Eventually, I think we'd want to move away from doing this and instead having a pre-generated datacube for these data collections, but in the meantime, that's the only use-case I can think of that blends those two function... so probably that's not a reason to avoid separating the functions either.

[TomNicholas]

Eventually, I think we'd want to move away from doing this and instead having a pre-generated datacube for these data collections

That was the use case VirtualiZarr was intended for, as opposed to runtime hackery 😁

that's the only use-case I can think of that blends those two function... so probably that's not a reason to avoid separating the functions either.

👍

[betolink]

Totally, however this runtime hackery has been very useful to access L3 datasets with dmrpp files available as if they had a pre-generated virtual store. I think we should keep supporting that IMO.

Now the load=True I think I remember Ayush telling me why that was necessary in our end (earthaccess) but I don't remember now. If I'm reading the code correctly, there is not an option to tell VirtualiZarr to only materialize the dimensions? so we had to translate the whole thing to Kerchunk to load that info? and I forgot why we needed that, maybe the slicing and index based operations xarray does?

[TomNicholas]

this runtime hackery has been very useful to access L3 datasets with dmrpp files available as if they had a pre-generated virtual store.

It does seem unreliable though - just because there are dmrpp files available doesn't mean that the files can be combined into one virtual store.

there is not an option to tell VirtualiZarr to only materialize the dimensions?

There is now, it's the default. The default behaviour of loadable_variables is now effectively the same as specifying
loadable_variables={ALL 1D COORDINATES WITH THE SAME NAME AS THEIR SOLE DIMENSION}. That's what you need to get an xarray object with indexes that are useful for .sel operations.

It was not always that way, so maybe Ayush was trying to get those indexes loaded?

[betolink]

It does seem unreliable though - just because there are dmrpp files available doesn't mean that the files can be combined into one virtual store.

It is unreliable 😄 but... this access pattern works great for a few relevant datasets and when it doesn't work it has helped us finding things we need to fix, e.g. inconsistent variable names, different compression codecs, etc.