Skip to content

Include all V2 codecs as filters#713

Merged
maxrjones merged 6 commits intomainfrom
simplify-v2-translation
Jul 21, 2025
Merged

Include all V2 codecs as filters#713
maxrjones merged 6 commits intomainfrom
simplify-v2-translation

Conversation

@maxrjones
Copy link
Member

This PR applies @d-v-b's suggestion from zarr-developers/numcodecs#767 (comment) to use the filters argument in ArrayV2Metadata to store all codecs, to avoid a regression where we truncate the number of BytesBytesCodecs.

  • Closes #xxxx
  • Tests added
  • Tests passing
  • Full type hint coverage
  • Changes are documented in docs/releases.rst
  • New functions/methods are listed in api.rst
  • New functionality has documentation

@codecov
Copy link

codecov bot commented Jul 20, 2025

Codecov Report

❌ Patch coverage is 84.61538% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.47%. Comparing base (6f5347d) to head (9b79194).
⚠️ Report is 37 commits behind head on main.

Files with missing lines Patch % Lines
virtualizarr/codecs.py 71.42% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #713      +/-   ##
==========================================
- Coverage   88.57%   87.47%   -1.11%     
==========================================
  Files          34       34              
  Lines        1821     1820       -1     
==========================================
- Hits         1613     1592      -21     
- Misses        208      228      +20     
Files with missing lines Coverage Δ
virtualizarr/parsers/hdf/hdf.py 95.34% <100.00%> (ø)
virtualizarr/parsers/kerchunk/translator.py 80.90% <100.00%> (ø)
virtualizarr/utils.py 67.79% <100.00%> (-3.42%) ⬇️
virtualizarr/codecs.py 94.54% <71.42%> (-3.42%) ⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment on lines +145 to +151
# TODO: Find a more robust way to exclude any bytes codecs
# TODO: Test round-tripping big endian since that is stored in the bytes codec in V3; it should be included in data type instead for V2
v2_codecs = [
_to_v2_codec(get_codec_config(codec))
for codec in v3_metadata.codecs
if codec.__class__.__name__ != "BytesCodec"
]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@d-v-b are all ArrayBytesCodecs irrelevant for Zarr V2 where we could exclude based on whether they are a ArrayBytesCodec instance?

@maxrjones maxrjones added this to the v2.0.0 milestone Jul 20, 2025
Copy link
Member

@TomNicholas TomNicholas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

I do think this NISAR test should never have been merged - a unit test that downloads 0.2GB of data is ridiculous. Then we should not need any concept of slow tests. But in the interests of expediency we can wait to swap that out in a different PR, as it's not an urgent issue.

@maxrjones maxrjones merged commit 3bf9f85 into main Jul 21, 2025
15 checks passed
@maxrjones maxrjones deleted the simplify-v2-translation branch July 21, 2025 01:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants