Manifest arrays use arrayv3metadata #429
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
abarciauskas-bgse
had a problem deploying
to
test-release
GitHub Actions
Failure
TomNicholas
reviewed
Feb 6, 2025
…not happy about this)
abarciauskas-bgse
had a problem deploying
to
test-release
GitHub Actions
Failure
abarciauskas-bgse
had a problem deploying
to
test-release
GitHub Actions
Failure
abarciauskas-bgse
had a problem deploying
to
test-release
GitHub Actions
Failure
abarciauskas-bgse
had a problem deploying
to
test-release
GitHub Actions
Failure
abarciauskas-bgse
had a problem deploying
to
test-release
GitHub Actions
Failure
abarciauskas-bgse
had a problem deploying
to
test-release
GitHub Actions
Failure
| A list of xarray variables. | ||
| """ | ||
| # This chunk determination logic mirrors zarr-python's create | ||
| # https://github.com/zarr-developers/zarr-python/blob/main/zarr/creation.py#L62-L66 |
Collaborator
Author
There was a problem hiding this comment.
I removed this comment because I think the reference is from a previous version of zarr-python - @sharkinsspatial do you know if we can and should include an updated link?
Collaborator
Author
|
@TomNicholas this PR is ready for re-review and 🤞🏽 hopefully good to merge (to
And then I might also create a ticket to test this branch against all (or most) current examples. |
TomNicholas
approved these changes
Feb 18, 2025
Member
TomNicholas
left a comment
TomNicholas
left a comment
There was a problem hiding this comment.
Thank you @abarciauskas-bgse ! Let's just get this merged into a stable branch so that others can build on it.
Collaborator
|
Awesome work @abarciauskas-bgse! |
This was referenced Feb 25, 2025
This was referenced Mar 6, 2025
Merged
maxrjones
pushed a commit
that referenced
this pull request
Mar 7, 2025
* Manifest arrays use arrayv3metadata (#429) * Added zarray_to_v3metadata and test * Working on manifest array tests * Fix test_manifests/test_array#TestConcat tests * Passing TestStack tests and add fixture * All test_manifests/test_array tests passing * Compressors should be list * Passing dmrpp tests * Passing test_hdf.py tests * Start to work on kerchunk tests * Add method to convert array v3 metadata to v2 metadata for kerchunk (not happy about this) * Fix fixtures and mark xfail netcdf3 * Test for convert_v3_to_v2_metadata * Deduplicate fixture for array v3 metadata * Parse filters and compressors from v3 metdata for v2 metadata * Rewrite extract_codecs * Refactor convert_to_codec_pipeline * Fix hdf integration tests * Test for convert_to_codec_pipeline * Refactor get_codecs and its tests * Fix most integration tests and writer tests * Fix xarray tests * Working on integration tests * Add expected type * Mark datetime tests xfail * Upgrade xarray for tests * xfail some unsupported zarr-python 3 data types * Require zarr * Remove zarr dep * import zarr, explicit dependency Co-authored-by: Tom Nicholas <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add zarr as a dependency * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Min numcodecs version * numcodecs>=0.15.1 in environment and upstream.yml conda env files * Working on mypy errors * Fix mypy errors and tests * Remove ZArray class * Just return metadata's shape * Create update metadata function * Fix typing for update_metadata * Check for regular chunk grid in manifest instantiation * Remove obsolete codecs code * Fix chunks function and add docstring * Remove custom zattrs type * Move some imports and make update_metadata a private method * Remove zarr.py * Add zarr to other ci env files * Fixture array_v3_metadata uses array_v3_metadata_dict * No need for union type for CodecPipeline * Use type alias * Add comment * Update virtualizarr/manifests/array_api.py Co-authored-by: Tom Nicholas <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revised copy_and_replace_metadata to be in utils and called correctly * Update virtualizarr/translators/kerchunk.py Co-authored-by: Tom Nicholas <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor create v3 array metadata * Rename to create_v3_array_metadata * Fix some codecs fixtures * Use global vars and simple fixture for creating codec pipelines * Remove redundant create_codec_pipeline fixture * Fix docstring * Use create_v3_array_metadata in from_kerchunk_refs * Add links to zarr-python 3.0 issues for big endian, datetime and timedelta data types * Reorganize conftest * Remove obsolete comment * Rename function numcodec_config_to_configurable * Fix parameters in docstring for convert_to_codec_pipeline * Revert change to pytest mark skipif for astropy * Remove commented arguments * Add classes to test_codecs and make zarr_array a fixture * Add tests for extract_codecs * Add test for get_codec_config * Remove obsolete comment * Add test for copy_and_replace_metadata * Add release notes * Attempt to fix rst links * Move convert_v3_to_v2_metadata to utils --------- Co-authored-by: Tom Nicholas <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is still very much a WIP - many tests and implementations still need to be fixed.
A few notes:
_parse_chunk_encoding_v3function since it is a private function and may change, which is why some of that logic is replicated inconvert_to_codec_pipelineChecklist
docs/releases.rstNew functions/methods are listed inapi.rstNew functionality has documentation