Skip to content

feat: Support variable chunking#877

Draft
maxrjones wants to merge 7 commits intozarr-developers:mainfrom
maxrjones:feature/rectilinear-chunk-grid
Draft

feat: Support variable chunking#877
maxrjones wants to merge 7 commits intozarr-developers:mainfrom
maxrjones:feature/rectilinear-chunk-grid

Conversation

@maxrjones
Copy link
Member

@maxrjones maxrjones commented Feb 17, 2026

Note: Mostly a proof of concept now to facilitate upstream reviews, but still opening for visibility

Overview

  • Add initial support for RectilinearChunkGrid (variable-length chunks), enabling VirtualiZarr to work with archival formats that have non-uniform chunk sizes (e.g., stripped TIFFs where image height isn't evenly divisible by rows per strip)
  • Remove the RegularChunkGrid-only restriction from ManifestArray and update the concatenation/stacking machinery to handle rectilinear chunk grids
  • Add a test that creates two stripped TIFFs with rectilinear chunks via virtual-tiff and concatenates them with xr.concat

Details

Core changes:

  • ManifestArray.chunks now returns chunk_grid.chunk_shapes (tuple of tuples) for RectilinearChunkGrid, rather than raising NotImplementedError. Uses the same compatibility pattern as xarray: check not isinstance(chunk_grid, RegularChunkGrid) behind a has_rectilinear_chunk_grid_support feature flag, so it falls back gracefully on older zarr versions
  • copy_and_replace_metadata detects nested chunk sequences (via vendored _is_nested_sequence) and creates the appropriate chunk grid type (regular vs rectilinear)
  • stack correctly inserts a single-element tuple (1,) when adding a new axis to a rectilinear array
  • check_no_partial_chunks_on_concat_axis skips the check for rectilinear grids, which explicitly encode variable chunk sizes
  • Vendored _is_nested_sequence from zarr-python#3534 into virtualizarr/vendor/zarr/core/chunk_grids.py for detecting rectilinear chunk representations

Compatibility:

  • All changes are backward-compatible with zarr 3.1.x (no RectilinearChunkGrid)
  • The test is skipped when zarr lacks RectilinearChunkGrid support
  • mypy passes cleanly on the standard test environment

Environment:

Checklist

@codecov
Copy link

codecov bot commented Feb 17, 2026

Codecov Report

❌ Patch coverage is 79.16667% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.93%. Comparing base (ba1f464) to head (bb7dc7e).

Files with missing lines Patch % Lines
virtualizarr/vendor/zarr/core/chunk_grids.py 71.42% 4 Missing ⚠️
virtualizarr/manifests/utils.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #877      +/-   ##
==========================================
- Coverage   89.11%   88.93%   -0.18%     
==========================================
  Files          34       35       +1     
  Lines        1956     1980      +24     
==========================================
+ Hits         1743     1761      +18     
- Misses        213      219       +6     
Files with missing lines Coverage Δ
virtualizarr/manifests/array.py 86.02% <100.00%> (+1.07%) ⬆️
virtualizarr/manifests/array_api.py 96.96% <100.00%> (+0.09%) ⬆️
virtualizarr/manifests/utils.py 90.47% <83.33%> (-0.27%) ⬇️
virtualizarr/vendor/zarr/core/chunk_grids.py 71.42% <71.42%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant