Skip to content

Adds a shards property to Array #2586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 46 commits into from
Closed

Adds a shards property to Array #2586

wants to merge 46 commits into from

Conversation

@normanrz
Copy link
Member

@normanrz normanrz commented Dec 25, 2024 •
edited
Loading

Builds on #2463

  • Adds a shards property to Array that returns the shard shape and None if sharding isn't used
  • chunks now returns the inner chunk shape, if sharding is used
  • Moves the chunks and shards properties into the ArrayMetadata classes to handle v2 and v3 cases

I was wondering if shards should equal chunks if sharding isn't used. Let me know what you think.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

d-v-b added 30 commits November 4, 2024 22:51
@d-v-b
add functions for easy read-only data access
fd6ecd1
@d-v-b
sync funcs
fa343f5
@d-v-b
make read-only funcs top-level exports
d95eba8
@d-v-b
Merge branch 'main' into feat/read-funcs
f6765bc
@d-v-b
add create_array, create_group, and tests
5d8445b
@d-v-b
add top-level imports
9526571
@d-v-b
Merge branch 'feat/read-funcs' of github.com:d-v-b/zarr-python into f…
90bf421 
…eat/read-funcs
@d-v-b
add test for top-level exports
de280a7
@d-v-b
add test for read
d9878cf
@d-v-b
add asserts
e5217ce
@d-v-b
Apply suggestions from code review
40cc7af
@d-v-b
Merge branch 'main' into feat/read-funcs
d7ce58b
@d-v-b
Merge branch 'main' into feat/read-funcs
a0dfe18
@d-v-b
Merge branch 'main' into feat/read-funcs
98bc328
@d-v-b
Merge branch 'main' of github.com:zarr-developers/zarr-python into fe…
16f5cc2 
…at/read-funcs
@d-v-b
handle sharding in create_array
4b45ebf
@d-v-b
Merge branch 'main' of github.com:zarr-developers/zarr-python into fe…
750a439 
…at/read-funcs
@d-v-b
tweak
7a5cbe7
@d-v-b
Merge branch 'main' of github.com:zarr-developers/zarr-python into fe…
215ff96 
…at/read-funcs
@d-v-b
make logic of _auto_partition better for shard shape
489e2a2
@d-v-b
add dtype parsing, and tweak auto_partitioning func
05dd0d8
@d-v-b
sketch of docstring; remove auto chunks / shard shape
3fbfc21
@d-v-b
Merge branch 'main' of github.com:zarr-developers/zarr-python into fe…
b348737 
…at/read-funcs
@d-v-b
tweak docstring
5025ad6
@d-v-b
Merge branch 'main' of github.com:zarr-developers/zarr-python into fe…
e204a32 
…at/read-funcs
@d-v-b
docstrings
68465db
@d-v-b
ensure tests pass
d7bb121
@d-v-b
tuple -> list
99cc8f5
@d-v-b
allow data in create_array
a39457f
@d-v-b
docstring
3f0a3e0
@normanrz normanrz requested a review from d-v-b December 25, 2024 20:18
@normanrz normanrz self-assigned this Dec 25, 2024
@d-v-b
Copy link
Contributor

d-v-b commented Dec 26, 2024

I was wondering if shards should equal chunks if sharding isn't used. Let me know what you think.

I think we should only report a tuple of ints for shards if and only if a sharding codec is being used -- otherwise it's not easy to tell the difference between an array that uses the sharding codec w/ shard shape=inner chunk shape, and an array that is not using the sharding codec. shards = None seems like a sane choice for the latter case.

@normanrz
Copy link
Member Author

normanrz commented Dec 27, 2024

Merged into #2463

@normanrz normanrz closed this Dec 27, 2024
@normanrz normanrz deleted the feat/chunks-shards branch December 27, 2024 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@d-v-b d-v-b d-v-b approved these changes

Assignees

@normanrz normanrz

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@normanrz @d-v-b @jhamman