Skip to content

Conversation

@maxrjones
Copy link
Member

This PR shows how to cache entire files before virtualizing.

Full explanation: for data formats that use b-link trees (like non-cloud-optimized version of HDF5), you'll probably want to cache the entire file up front. If you also want to load many variables, you'll want to cache the file's contents at the store level rather than at the parser/file reader level so that it's accessible by ManifestStore as well as the parser. This PR adds an example of how to do that using new features in obspec-utils. It relies on the sequence of PRs discussed in #844, which makes VirtualiZarr work with any ReadableStore using duck-type rather than only the stores implemented in obstore.

It reduces the amount of time to virtualize a single GOES-16 file on my laptop from ~47s to ~8s ⚡

  • Closes #xxxx
  • Tests added
  • Tests passing
  • Full type hint coverage
  • Changes are documented in docs/releases.rst
  • New functions/methods are listed in api.rst
  • New functionality has documentation

@codecov
Copy link

codecov bot commented Jan 24, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.89%. Comparing base (2bbd1f9) to head (a4aee66).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #855      +/-   ##
==========================================
- Coverage   89.33%   88.89%   -0.44%     
==========================================
  Files          34       34              
  Lines        2015     1945      -70     
==========================================
- Hits         1800     1729      -71     
- Misses        215      216       +1     

see 13 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TomNicholas TomNicholas merged commit 6f1b4d7 into zarr-developers:main Jan 24, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants