Skip to content

Commit 036fd2a

Browse files
d-v-bd-v-b
committed
Merge branch 'main' of github.com:zarr-developers/zarr-python into feat/batch-creation
2 parents cdfd5de + 45146ca commit 036fd2a

File tree

12 files changed

+145
-54
lines changed

12 files changed

+145
-54
lines changed

‎.github/workflows/test.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,12 @@ jobs:
6060
hatch env run -e test.py${{ matrix.python-version }}-${{ matrix.numpy-version }}-${{ matrix.dependency-set }} list-env
6161
- name: Run Tests
6262
run: |
63-
hatch env run --env test.py${{ matrix.python-version }}-${{ matrix.numpy-version }}-${{ matrix.dependency-set }} run
63+
hatch env run --env test.py${{ matrix.python-version }}-${{ matrix.numpy-version }}-${{ matrix.dependency-set }} run-coverage
64+
- name: Upload coverage
65+
uses: codecov/codecov-action@v5
66+
with:
67+
token: ${{ secrets.CODECOV_TOKEN }}
68+
verbose: true # optional (default = false)
6469

6570
test-upstream-and-min-deps:
6671
name: py=${{ matrix.python-version }}-${{ matrix.dependency-set }}

‎codecov.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
coverage:
2+
status:
3+
patch:
4+
default:
5+
target: auto
6+
project:
7+
default:
8+
target: auto
9+
threshold: 0.1
10+
comment: false

‎docs/developers/contributing.rst‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -329,10 +329,16 @@ Release procedure
329329
Most of the release process is now handled by GitHub workflow which should
330330
automatically push a release to PyPI if a tag is pushed.
331331

332-
Before releasing, make sure that all pull requests which will be
333-
included in the release have been properly documented in
334-
`docs/release.rst`.
335-
332+
Pre-release
333+
"""""""""""
334+
1. Make sure that all pull requests which will be
335+
included in the release have been properly documented in
336+
:file:`docs/release-notes.rst`.
337+
2. Rename the "Unreleased" section heading in :file:`docs/release-notes.rst`
338+
to the version you are about to release.
339+
340+
Releasing
341+
"""""""""
336342
To make a new release, go to
337343
https://github.com/zarr-developers/zarr-python/releases and
338344
click "Draft a new release". Choose a version number prefixed
@@ -355,5 +361,8 @@ https://readthedocs.io. Full releases will be available under
355361
pre-releases will be available under
356362
`/latest <https://zarr.readthedocs.io/en/latest>`_.
357363

358-
Also review and merge the https://github.com/conda-forge/zarr-feedstock
359-
pull request that will be automatically generated.
364+
Post-release
365+
""""""""""""
366+
367+
- Review and merge the pull request on the `conda-forge feedstock <https://github.com/conda-forge/zarr-feedstock>`_ that will be automatically generated.
368+
- Create a new "Unreleased" section in the release notes

‎docs/quickstart.rst‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@ Zarr supports data compression and filters. For example, to use Blosc compressio
7474
... "data/example-3.zarr",
7575
... mode="w", shape=(100, 100),
7676
... chunks=(10, 10), dtype="f4",
77-
... compressor=zarr.codecs.BloscCodec(cname="zstd", clevel=3, shuffle=zarr.codecs.BloscShuffle.SHUFFLE)
77+
... compressors=zarr.codecs.BloscCodec(cname="zstd", clevel=3, shuffle=zarr.codecs.BloscShuffle.shuffle)
7878
... )
7979
>>> z[:, :] = np.random.random((100, 100))
8080
>>>
@@ -101,7 +101,7 @@ Zarr allows you to create hierarchical groups, similar to directories::
101101
>>> root = zarr.group("data/example-2.zarr")
102102
>>> foo = root.create_group(name="foo")
103103
>>> bar = root.create_array(
104-
... name="bar", shape=(100, 10), chunks=(10, 10)
104+
... name="bar", shape=(100, 10), chunks=(10, 10), dtype="f4"
105105
... )
106106
>>> spam = foo.create_array(name="spam", shape=(10,), dtype="i4")
107107
>>>
@@ -112,6 +112,7 @@ Zarr allows you to create hierarchical groups, similar to directories::
112112
>>> # print the hierarchy
113113
>>> root.tree()
114114
/
115+
├── bar (100, 10) float32
115116
└── foo
116117
└── spam (10,) int32
117118
<BLANKLINE>
@@ -130,7 +131,7 @@ using external libraries like `s3fs <https://s3fs.readthedocs.io>`_ or
130131

131132
>>> import s3fs # doctest: +SKIP
132133
>>>
133-
>>> z = zarr.create_array("s3://example-bucket/foo", mode="w", shape=(100, 100), chunks=(10, 10)) # doctest: +SKIP
134+
>>> z = zarr.create_array("s3://example-bucket/foo", mode="w", shape=(100, 100), chunks=(10, 10), dtype="f4") # doctest: +SKIP
134135
>>> z[:, :] = np.random.random((100, 100)) # doctest: +SKIP
135136

136137
A single-file store can also be created using the the :class:`zarr.storage.ZipStore`::

‎docs/release-notes.rst‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,21 @@ New features
99

1010
Bug fixes
1111
~~~~~~~~~
12-
* Fixes ``order`` argument for Zarr format 2 arrays.
13-
By :user:`Norman Rzepka <normanrz>` (:issue:`2679`).
12+
* Fixes ``order`` argument for Zarr format 2 arrays (:issue:`2679`).
13+
14+
* Fixes a bug that prevented reading Zarr format 2 data with consolidated metadata written using ``zarr-python`` version 2 (:issue:`2694`).
15+
16+
* Ensure that compressor=None results in no compression when writing Zarr format 2 data (:issue:`2708`)
1417

1518
Behaviour changes
1619
~~~~~~~~~~~~~~~~~
1720

21+
Other
22+
~~~~~
23+
* Removed some unnecessary files from the source distribution
24+
to reduce its size. (:issue:`2686`)
25+
26+
1827
.. _release_3.0.0:
1928

2029
3.0.0

‎pyproject.toml‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,13 @@
22
requires = ["hatchling", "hatch-vcs"]
33
build-backend = "hatchling.build"
44

5+
[tool.hatch.build.targets.sdist]
6+
exclude = [
7+
"/.github",
8+
"/bench",
9+
"/docs",
10+
"/notebooks"
11+
]
512

613
[project]
714
name = "zarr"
@@ -103,13 +110,13 @@ Homepage = "https://github.com/zarr-developers/zarr-python"
103110
[tool.coverage.report]
104111
exclude_lines = [
105112
"pragma: no cover",
113+
"if TYPE_CHECKING:",
106114
"pragma: ${PY_MAJOR_VERSION} no cover",
107115
'.*\.\.\.' # Ignore "..." lines
108116
]
109117

110118
[tool.coverage.run]
111119
omit = [
112-
"src/zarr/meta_v1.py",
113120
"bench/compress_normal.py",
114121
]
115122

@@ -140,8 +147,8 @@ numpy = ["1.25", "2.1"]
140147
features = ["gpu"]
141148

142149
[tool.hatch.envs.test.scripts]
143-
run-coverage = "pytest --cov-config=pyproject.toml --cov=pkg --cov=src"
144-
run-coverage-gpu = "pip install cupy-cuda12x && pytest -m gpu --cov-config=pyproject.toml --cov=pkg --cov=src"
150+
run-coverage = "pytest --cov-config=pyproject.toml --cov=pkg --cov-report xml --cov=src --junitxml=junit.xml -o junit_family=legacy"
151+
run-coverage-gpu = "pip install cupy-cuda12x && pytest -m gpu --cov-config=pyproject.toml --cov=pkg --cov-report xml --cov=src --junitxml=junit.xml -o junit_family=legacy"
145152
run = "run-coverage --no-cov"
146153
run-verbose = "run-coverage --verbose"
147154
run-mypy = "mypy src"
@@ -170,7 +177,7 @@ numpy = ["1.25", "2.1"]
170177
version = ["minimal"]
171178

172179
[tool.hatch.envs.gputest.scripts]
173-
run-coverage = "pytest -m gpu --cov-config=pyproject.toml --cov=pkg --cov=src"
180+
run-coverage = "pytest -m gpu --cov-config=pyproject.toml --cov=pkg --cov-report xml --cov=src --junitxml=junit.xml -o junit_family=legacy"
174181
run = "run-coverage --no-cov"
175182
run-verbose = "run-coverage --verbose"
176183
run-mypy = "mypy src"

‎src/zarr/core/array.py‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4131,15 +4131,22 @@ def _parse_chunk_encoding_v3(
41314131

41324132

41334133
def _parse_deprecated_compressor(
4134-
compressor: CompressorLike | None, compressors: CompressorsLike
4134+
compressor: CompressorLike | None, compressors: CompressorsLike, zarr_format: int = 3
41354135
) -> CompressorsLike | None:
4136-
if compressor:
4136+
if compressor != "auto":
41374137
if compressors != "auto":
41384138
raise ValueError("Cannot specify both `compressor` and `compressors`.")
4139-
warn(
4140-
"The `compressor` argument is deprecated. Use `compressors` instead.",
4141-
category=UserWarning,
4142-
stacklevel=2,
4143-
)
4144-
compressors = (compressor,)
4139+
if zarr_format == 3:
4140+
warn(
4141+
"The `compressor` argument is deprecated. Use `compressors` instead.",
4142+
category=UserWarning,
4143+
stacklevel=2,
4144+
)
4145+
if compressor is None:
4146+
# "no compression"
4147+
compressors = ()
4148+
else:
4149+
compressors = (compressor,)
4150+
elif zarr_format == 2 and compressor == compressors == "auto":
4151+
compressors = ({"id": "blosc"},)
41454152
return compressors

‎src/zarr/core/group.py‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -594,8 +594,8 @@ def _from_bytes_v2(
594594
v2_consolidated_metadata = json.loads(consolidated_metadata_bytes.to_bytes())
595595
v2_consolidated_metadata = v2_consolidated_metadata["metadata"]
596596
# We already read zattrs and zgroup. Should we ignore these?
597-
v2_consolidated_metadata.pop(".zattrs")
598-
v2_consolidated_metadata.pop(".zgroup")
597+
v2_consolidated_metadata.pop(".zattrs", None)
598+
v2_consolidated_metadata.pop(".zgroup", None)
599599

600600
consolidated_metadata: defaultdict[str, dict[str, Any]] = defaultdict(dict)
601601

@@ -1032,7 +1032,7 @@ async def create_array(
10321032
shards: ShardsLike | None = None,
10331033
filters: FiltersLike = "auto",
10341034
compressors: CompressorsLike = "auto",
1035-
compressor: CompressorLike = None,
1035+
compressor: CompressorLike = "auto",
10361036
serializer: SerializerLike = "auto",
10371037
fill_value: Any | None = 0,
10381038
order: MemoryOrder | None = None,
@@ -1135,8 +1135,9 @@ async def create_array(
11351135
AsyncArray
11361136
11371137
"""
1138-
1139-
compressors = _parse_deprecated_compressor(compressor, compressors)
1138+
compressors = _parse_deprecated_compressor(
1139+
compressor, compressors, zarr_format=self.metadata.zarr_format
1140+
)
11401141
return await create_array(
11411142
store=self.store_path,
11421143
name=name,
@@ -2329,7 +2330,7 @@ def create_array(
23292330
shards: ShardsLike | None = None,
23302331
filters: FiltersLike = "auto",
23312332
compressors: CompressorsLike = "auto",
2332-
compressor: CompressorLike = None,
2333+
compressor: CompressorLike = "auto",
23332334
serializer: SerializerLike = "auto",
23342335
fill_value: Any | None = 0,
23352336
order: MemoryOrder | None = "C",
@@ -2431,7 +2432,9 @@ def create_array(
24312432
-------
24322433
AsyncArray
24332434
"""
2434-
compressors = _parse_deprecated_compressor(compressor, compressors)
2435+
compressors = _parse_deprecated_compressor(
2436+
compressor, compressors, zarr_format=self.metadata.zarr_format
2437+
)
24352438
return Array(
24362439
self._sync(
24372440
self._async_group.create_array(

‎tests/test_group.py‎

Lines changed: 2 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111

1212
import numpy as np
1313
import pytest
14-
from numcodecs import Zstd
14+
from numcodecs import Blosc
1515

1616
import zarr
1717
import zarr.api.asynchronous
@@ -507,7 +507,7 @@ def test_group_child_iterators(store: Store, zarr_format: ZarrFormat, consolidat
507507
"chunks": (1,),
508508
"order": "C",
509509
"filters": None,
510-
"compressor": Zstd(level=0),
510+
"compressor": Blosc(),
511511
"zarr_format": zarr_format,
512512
},
513513
"subgroup": {
@@ -1678,13 +1678,3 @@ def test_group_members_concurrency_limit(store: MemoryStore) -> None:
16781678
elapsed = time.time() - start
16791679

16801680
assert elapsed > num_groups * get_latency
1681-
1682-
1683-
@pytest.mark.parametrize("store", ["local", "memory"], indirect=["store"])
1684-
def test_deprecated_compressor(store: Store) -> None:
1685-
g = zarr.group(store=store, zarr_format=2)
1686-
with pytest.warns(UserWarning, match="The `compressor` argument is deprecated.*"):
1687-
a = g.create_array(
1688-
"foo", shape=(100,), chunks=(10,), dtype="i4", compressor={"id": "blosc"}
1689-
)
1690-
assert a.metadata.compressor.codec_id == "blosc"

‎tests/test_metadata/test_consolidated.py‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55

66
import numpy as np
77
import pytest
8-
from numcodecs import Zstd
8+
from numcodecs import Blosc
99

1010
import zarr.api.asynchronous
1111
import zarr.api.synchronous
@@ -17,7 +17,7 @@
1717
open,
1818
open_consolidated,
1919
)
20-
from zarr.core.buffer import default_buffer_prototype
20+
from zarr.core.buffer import cpu, default_buffer_prototype
2121
from zarr.core.group import ConsolidatedMetadata, GroupMetadata
2222
from zarr.core.metadata import ArrayV3Metadata
2323
from zarr.core.metadata.v2 import ArrayV2Metadata
@@ -476,6 +476,30 @@ async def test_open_consolidated_raises_async(self, zarr_format: ZarrFormat):
476476
with pytest.raises(ValueError):
477477
await zarr.api.asynchronous.open_consolidated(store, zarr_format=None)
478478

479+
@pytest.fixture
480+
async def v2_consolidated_metadata_empty_dataset(
481+
self, memory_store: zarr.storage.MemoryStore
482+
) -> AsyncGroup:
483+
zgroup_bytes = cpu.Buffer.from_bytes(json.dumps({"zarr_format": 2}).encode())
484+
zmetadata_bytes = cpu.Buffer.from_bytes(
485+
b'{"metadata":{".zgroup":{"zarr_format":2}},"zarr_consolidated_format":1}'
486+
)
487+
return AsyncGroup._from_bytes_v2(
488+
None, zgroup_bytes, zattrs_bytes=None, consolidated_metadata_bytes=zmetadata_bytes
489+
)
490+
491+
async def test_consolidated_metadata_backwards_compatibility(
492+
self, v2_consolidated_metadata_empty_dataset
493+
):
494+
"""
495+
Test that consolidated metadata handles a missing .zattrs key. This is necessary for backwards compatibility with zarr-python 2.x. See https://github.com/zarr-developers/zarr-python/issues/2694
496+
"""
497+
store = zarr.storage.MemoryStore()
498+
await zarr.api.asynchronous.open(store=store, zarr_format=2)
499+
await zarr.api.asynchronous.consolidate_metadata(store)
500+
result = await zarr.api.asynchronous.open_consolidated(store, zarr_format=2)
501+
assert result.metadata == v2_consolidated_metadata_empty_dataset.metadata
502+
479503
async def test_consolidated_metadata_v2(self):
480504
store = zarr.storage.MemoryStore()
481505
g = await AsyncGroup.from_store(store, attributes={"key": "root"}, zarr_format=2)
@@ -498,7 +522,7 @@ async def test_consolidated_metadata_v2(self):
498522
attributes={"key": "a"},
499523
chunks=(1,),
500524
fill_value=0,
501-
compressor=Zstd(level=0),
525+
compressor=Blosc(),
502526
order="C",
503527
),
504528
"g1": GroupMetadata(

0 commit comments

Comments
 (0)