Skip to content

Manifest arrays use arrayv3metadata #429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
abarciauskas-bgse merged 81 commits into from
Feb 18, 2025
Merged

Manifest arrays use arrayv3metadata #429

Show file tree
Hide file tree
Changes from 80 commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
2a01bfa
Added zarray_to_v3metadata and test
abarciauskas-bgse Feb 4, 2025
17fd547
Working on manifest array tests
abarciauskas-bgse Feb 5, 2025
e5666ab
Fix test_manifests/test_array#TestConcat tests
abarciauskas-bgse Feb 5, 2025
5a8cc4c
Passing TestStack tests and add fixture
abarciauskas-bgse Feb 5, 2025
4c0b616
All test_manifests/test_array tests passing
abarciauskas-bgse Feb 6, 2025
ac2f787
Compressors should be list
abarciauskas-bgse Feb 6, 2025
5503c60
Passing dmrpp tests
abarciauskas-bgse Feb 6, 2025
1272051
Merge branch 'main' into manifest-arrays-use-arrayv3metadata
abarciauskas-bgse Feb 6, 2025
1f36755
Passing test_hdf.py tests
abarciauskas-bgse Feb 6, 2025
7098803
Start to work on kerchunk tests
abarciauskas-bgse Feb 6, 2025
ce2284c
Add method to convert array v3 metadata to v2 metadata for kerchunk (…
abarciauskas-bgse Feb 7, 2025
c9853d5
Fix fixtures and mark xfail netcdf3
abarciauskas-bgse Feb 7, 2025
209dae3
Test for convert_v3_to_v2_metadata
abarciauskas-bgse Feb 7, 2025
e7205ef
Deduplicate fixture for array v3 metadata
abarciauskas-bgse Feb 7, 2025
d65e457
Parse filters and compressors from v3 metdata for v2 metadata
abarciauskas-bgse Feb 7, 2025
190c20f
Rewrite extract_codecs
abarciauskas-bgse Feb 7, 2025
47f5ddd
Refactor convert_to_codec_pipeline
abarciauskas-bgse Feb 8, 2025
5d15608
Fix hdf integration tests
abarciauskas-bgse Feb 8, 2025
908bc52
Test for convert_to_codec_pipeline
abarciauskas-bgse Feb 8, 2025
4a8bfdd
Refactor get_codecs and its tests
abarciauskas-bgse Feb 8, 2025
d05cec3
Fix most integration tests and writer tests
abarciauskas-bgse Feb 9, 2025
ff23eeb
Fix xarray tests
abarciauskas-bgse Feb 9, 2025
8560f2d
Working on integration tests
abarciauskas-bgse Feb 9, 2025
97d0a71
Add expected type
abarciauskas-bgse Feb 10, 2025
669ce52
Mark datetime tests xfail
abarciauskas-bgse Feb 10, 2025
b794dab
Upgrade xarray for tests
abarciauskas-bgse Feb 10, 2025
825142d
xfail some unsupported zarr-python 3 data types
abarciauskas-bgse Feb 10, 2025
6684125
Require zarr
abarciauskas-bgse Feb 10, 2025
5e82de4
Remove zarr dep
abarciauskas-bgse Feb 10, 2025
f57b48d
import zarr, explicit dependency
abarciauskas-bgse Feb 10, 2025
b811959
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 10, 2025
8c5139b
Add zarr as a dependency
abarciauskas-bgse Feb 11, 2025
eb2a86c
Merge branch 'manifest-arrays-use-arrayv3metadata' of github.com:zarr…
abarciauskas-bgse Feb 11, 2025
15ac7a7
Merge branch 'main' into manifest-arrays-use-arrayv3metadata
abarciauskas-bgse Feb 11, 2025
5359762
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 11, 2025
c808351
Min numcodecs version
abarciauskas-bgse Feb 12, 2025
bd50167
numcodecs>=0.15.1 in environment and upstream.yml conda env files
abarciauskas-bgse Feb 12, 2025
ed97704
Working on mypy errors
abarciauskas-bgse Feb 12, 2025
a3c190e
Fix mypy errors and tests
abarciauskas-bgse Feb 12, 2025
95886b9
Remove ZArray class
abarciauskas-bgse Feb 12, 2025
a0f72b2
Just return metadata's shape
abarciauskas-bgse Feb 12, 2025
aad511f
Create update metadata function
abarciauskas-bgse Feb 12, 2025
b357b04
Fix typing for update_metadata
abarciauskas-bgse Feb 12, 2025
08e877a
Check for regular chunk grid in manifest instantiation
abarciauskas-bgse Feb 12, 2025
f040459
Remove obsolete codecs code
abarciauskas-bgse Feb 12, 2025
495d660
Fix chunks function and add docstring
abarciauskas-bgse Feb 12, 2025
a262f0b
Remove custom zattrs type
abarciauskas-bgse Feb 12, 2025
bcd68a0
Move some imports and make update_metadata a private method
abarciauskas-bgse Feb 12, 2025
f0ce778
Remove zarr.py
abarciauskas-bgse Feb 12, 2025
0518488
Add zarr to other ci env files
abarciauskas-bgse Feb 13, 2025
0712979
Fixture array_v3_metadata uses array_v3_metadata_dict
abarciauskas-bgse Feb 13, 2025
c40915d
No need for union type for CodecPipeline
abarciauskas-bgse Feb 13, 2025
cdaca53
Use type alias
abarciauskas-bgse Feb 13, 2025
2415e07
Add comment
abarciauskas-bgse Feb 13, 2025
9366d69
Update virtualizarr/manifests/array_api.py
abarciauskas-bgse Feb 13, 2025
d590cfc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 13, 2025
6394207
Revised copy_and_replace_metadata to be in utils and called correctly
abarciauskas-bgse Feb 13, 2025
ea9fd56
Update virtualizarr/translators/kerchunk.py
abarciauskas-bgse Feb 13, 2025
0ee2b48
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Feb 13, 2025
86d1de5
Refactor create v3 array metadata
abarciauskas-bgse Feb 13, 2025
fe8305f
Rename to create_v3_array_metadata
abarciauskas-bgse Feb 13, 2025
0f5b32d
Fix some codecs fixtures
abarciauskas-bgse Feb 13, 2025
97bc7cd
Use global vars and simple fixture for creating codec pipelines
abarciauskas-bgse Feb 13, 2025
b5a1dc6
Remove redundant create_codec_pipeline fixture
abarciauskas-bgse Feb 13, 2025
12c6260
Fix docstring
abarciauskas-bgse Feb 13, 2025
4b555b6
Use create_v3_array_metadata in from_kerchunk_refs
abarciauskas-bgse Feb 13, 2025
c245b0a
Add links to zarr-python 3.0 issues for big endian, datetime and time…
abarciauskas-bgse Feb 14, 2025
23ac776
Reorganize conftest
abarciauskas-bgse Feb 14, 2025
402af7f
Remove obsolete comment
abarciauskas-bgse Feb 15, 2025
3e2e8ec
Rename function numcodec_config_to_configurable
abarciauskas-bgse Feb 15, 2025
92293a6
Fix parameters in docstring for convert_to_codec_pipeline
abarciauskas-bgse Feb 15, 2025
eccc215
Revert change to pytest mark skipif for astropy
abarciauskas-bgse Feb 15, 2025
8fa72b2
Remove commented arguments
abarciauskas-bgse Feb 15, 2025
f747681
Add classes to test_codecs and make zarr_array a fixture
abarciauskas-bgse Feb 15, 2025
ba2e467
Add tests for extract_codecs
abarciauskas-bgse Feb 16, 2025
68695c9
Add test for get_codec_config
abarciauskas-bgse Feb 16, 2025
f22bf96
Remove obsolete comment
abarciauskas-bgse Feb 17, 2025
8af37a0
Add test for copy_and_replace_metadata
abarciauskas-bgse Feb 17, 2025
d4946ab
Add release notes
abarciauskas-bgse Feb 17, 2025
2c43c61
Attempt to fix rst links
abarciauskas-bgse Feb 17, 2025
05cd57a
Move convert_v3_to_v2_metadata to utils
abarciauskas-bgse Feb 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions ci/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ dependencies:
- packaging
- universal_pathlib
- hdf5plugin
- numcodecs
- numcodecs>=0.15.1
- icechunk>=0.1.1
# Testing
- codecov[toml]
Expand All @@ -36,5 +36,6 @@ dependencies:
# for opening FITS files
- astropy
- pip
- zarr>=3.0.2
- pip:
- imagecodecs-numcodecs==2024.6.1
- imagecodecs-numcodecs==2024.6.1
5 changes: 3 additions & 2 deletions ci/min-deps.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ dependencies:
- h5py
- hdf5
- netcdf4
- xarray>=2024.10.0
- xarray>=2025.1.1
- numpy>=2.0.0
- numcodecs
- numcodecs>=0.15.1
- packaging
- ujson
- universal_pathlib
Expand All @@ -23,3 +23,4 @@ dependencies:
- pytest
- pooch
- fsspec
- zarr>=3.0.2
3 changes: 2 additions & 1 deletion ci/upstream.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ dependencies:
- ujson
- universal_pathlib
- hdf5plugin
- numcodecs
- numcodecs>=0.15.1
- imagecodecs>=2024.6.1
# Testing
- codecov[toml]
Expand All @@ -28,6 +28,7 @@ dependencies:
- pooch
- fsspec
- pip
- zarr>=3.0.2
- pip:
- git+https://github.com/earth-mover/icechunk.git@main#subdirectory=icechunk-python # Installs zarr-python v3.0.0 as dependency
- git+https://github.com/fsspec/kerchunk.git@main
Expand Down
Loading