Feasibility Report: Virtual Stores for NASA

[abarciauskas-bgse]

We want to support adoption by ESDIS by reporting:

1. Why they would want to use this technology (home page)
   1. What do virtual stores enable?
   2. A vision for a future where all datasets are accessible through a single entrypoint
   3. How does it help data users?
   4. How does it help data providers and NASA overall save money?
2. Technical Overview
3. Application of Virtual Store Technology to NASA Data
   1. Mature
      a. Consistently gridded data into collection-level aggregated chunk manifests (PODAAC)
   2. Developing
      b. Displacement data (ASF)
      c. Ongoing data: Appending via icechunk or overwriting kerchunk JSON
      a. different grids and compression schemes, L2/orbital swath data
   3. What will not be supported
4. Known limitations:
   1. Icechunk is very python + rust centric
   2. Structural decisions about data formatting, chunking and chunk manifests made early on impact performance for different use cases. Still cannot simultaneously optimize for all use cases. Ideally, chunk manifests could be aggregated dynamically depending on the use case. For NISAR, for example, the current design for chunk manifests assumes users will be working with frames and thus optimizes for loading a chunk manifest per frame, but then you cannot easily load across frames.
5. Governance decisions which need to be made
   1. Standards for where to put the metadata - collection-level and frame-level
6. Established Best practices
   1. Typical use case patterns should be considered when designing the files/chunks and aggregated chunk manifests. For example, frames with NISAR data because of typical time series analysis.

This report would also include a link to:

1. An onboarding guide for anyone who is looking to get started virtualizing data.
2. An initial library of virtualization examples. These examples would represent the variety of data and use case patterns which have already been solved and serve as a resource for virtual layer producers.

A few more important points to be included:

• Adoption of GeoZarr and multiscales (recommendations)
• Chunk manifest protocol is in progress

[maxrjones]

I think it's worth including a comparison to DMR++ and Kerchunk approaches. I was a bit concerned about the current strategy of producing many more DMR++ files as a pathway to virtual datacubes since DMR++ do not currently include e-tag information and therefore provide no guarantees about consistency with the raw data.

[abarciauskas-bgse]

Noting that we should use https://nasa-impact.github.io/dse-virtual-zarr-workshop as a starting point for an onboarding guide and https://github.com/virtual-zarr/virtual-zarr-cookbook/issues/1#issuecomment-3427301497 has links to a lot of potentially relevant resources.

[sharkinsspatial]

@maxrjones @abarciauskas-bgse 👋 I’m stuck in a waiting room so I thought I might chime in 😆. One thing that might be worth emphasizing or clarifying is the distinction between individual granule virtualization and assembling data cubes. IIRC from our discussions with Owen and team, there seemed to be a focus on simply virtualizing individual granule access via DMRPP (or Virtualizarr).

I think it’s important to reinforce the true value of virtual data cubes (referencing entire collections) as a high performance index which provides a scalable, serverless alternative to metadata severs like STAC and CMR.

It feels like there may still be a bit of a communication gap around this point that might be obscuring the full value of Virtualization.

[abarciauskas-bgse]

We should also include that integration with EGIS as a thing we are interested in supporting with these virtual stores. I'm waiting to hear if we can work with their technical team to assess if that is feasible.

[abarciauskas-bgse]

A few more important points to be included:

• Adoption of GeoZarr and multiscales (recommendations)
• Chunk manifest protocol