Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
5483956
add creation from other zarr
brokkoli71 Jan 2, 2025
9a32d1f
remove duplicated tests
brokkoli71 Jan 2, 2025
2c19072
improve test
brokkoli71 Jan 2, 2025
91152be
test_iter_grid for non-squares
brokkoli71 Jan 2, 2025
3c5ec3f
concurrent streaming for equal chunk sizes
brokkoli71 Jan 8, 2025
a675475
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Jan 8, 2025
79a45b1
fix merge
brokkoli71 Jan 8, 2025
da2f03f
fix mypy
brokkoli71 Jan 8, 2025
a7553a7
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Jan 8, 2025
7728d7f
fix mypy
brokkoli71 Jan 8, 2025
2df18a0
fix test_iter_grid
brokkoli71 Jan 8, 2025
03e2500
extract to zarr.from_array
brokkoli71 Jan 8, 2025
f6ae2f8
fix mypy
brokkoli71 Jan 8, 2025
36146e5
fix mypy
brokkoli71 Jan 8, 2025
085efe9
format
brokkoli71 Jan 9, 2025
353a477
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Jan 9, 2025
93ed8d6
fix test_creation_from_other_zarr_format
brokkoli71 Jan 9, 2025
e607105
distinguish between keep and auto for from_array arguments
brokkoli71 Jan 15, 2025
7eb6988
partition concurrency along new_array chunks
brokkoli71 Jan 15, 2025
c7393a4
fix mypy
brokkoli71 Jan 15, 2025
543099a
improve test_creation_from_other_zarr_format
brokkoli71 Jan 15, 2025
73843dc
add typing in test
brokkoli71 Jan 15, 2025
0f0f812
Update src/zarr/core/array.py
brokkoli71 Jan 15, 2025
97b7b9b
Merge remote-tracking branch 'origin/creation-from-other-zarr' into c…
brokkoli71 Jan 15, 2025
021ca95
add from_array with npt.ArrayLike
brokkoli71 Jan 15, 2025
092a1e0
add write_data argument
brokkoli71 Jan 15, 2025
58f05fe
improve tests
brokkoli71 Jan 15, 2025
a4b4456
improve docstrings and add examples
brokkoli71 Jan 15, 2025
fc69b67
fix mypy and readthedocs
brokkoli71 Jan 15, 2025
4f3e156
fix mypy and readthedocs
brokkoli71 Jan 15, 2025
addb785
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Jan 15, 2025
a2373ad
fix mypy and readthedocs
brokkoli71 Jan 15, 2025
2706f17
fix mypy and readthedocs
brokkoli71 Jan 15, 2025
d5ccda1
fix readthedocs ERROR: Unexpected indentation
brokkoli71 Jan 15, 2025
e3d691d
add release notes
brokkoli71 Jan 15, 2025
29074fd
format docstring examples
brokkoli71 Jan 15, 2025
4ad7a5a
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Jan 30, 2025
39e05e9
add write_data attr to synchronous.create_array
brokkoli71 Jan 31, 2025
f15f5ac
`create_array` calls `from_array` calls `init_array`
brokkoli71 Jan 31, 2025
578469f
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Jan 31, 2025
c8c7b57
document changes
brokkoli71 Jan 31, 2025
b2aa207
fix serializer from_array v2 to v3
brokkoli71 Jan 31, 2025
a9f7f11
fix mypy
brokkoli71 Feb 1, 2025
e22db34
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Feb 1, 2025
7a285ff
improve codecov
brokkoli71 Feb 1, 2025
4e26cca
fix mypy
brokkoli71 Feb 1, 2025
c5c8875
from_array: copy zarr format on default
brokkoli71 Feb 1, 2025
92702a9
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Feb 5, 2025
f17df89
in ``from_array`` make all arguments except ``store`` keyword-only, t…
brokkoli71 Feb 5, 2025
826d6bd
in ``from_array`` default shards="keep"
brokkoli71 Feb 5, 2025
edd4852
redundant ``ChunkKeyEncoding | ChunkKeyEncodingLike``
brokkoli71 Feb 5, 2025
38aee40
fix argument order in calls of `from_array`
brokkoli71 Feb 5, 2025
6725cca
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Feb 5, 2025
c1e972d
fix numpydoc-validation
brokkoli71 Feb 5, 2025
df7a8ee
add docstring to store2 pytest fixture
brokkoli71 Feb 5, 2025
2e6dab2
extract `_parse_keep_array_attr` from `from_array`
brokkoli71 Feb 5, 2025
9782aed
extract `_parse_keep_array_attr` from `from_array`
brokkoli71 Feb 5, 2025
a698e54
Merge remote-tracking branch 'origin/creation-from-other-zarr' into c…
brokkoli71 Feb 5, 2025
c499b78
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Feb 5, 2025
aad5ca1
correct _parse_keep_array_attr
brokkoli71 Feb 5, 2025
c5d6f5b
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Feb 9, 2025
30aae3a
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Feb 11, 2025
0da7105
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Feb 15, 2025
6db5e0b
fix merge
brokkoli71 Feb 15, 2025
888b984
fix merge
brokkoli71 Feb 15, 2025
50375b6
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Feb 18, 2025
89e6c37
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Feb 28, 2025
2d88f86
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Apr 2, 2025
57d54a5
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Apr 7, 2025
8436218
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Apr 10, 2025
8de5baf
Merge branch 'main' into creation-from-other-zarr
brokkoli71 Apr 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/zarr/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
create_group,
empty,
empty_like,
from_array,
full,
full_like,
group,
Expand Down Expand Up @@ -52,6 +53,7 @@
"create_group",
"empty",
"empty_like",
"from_array",
"full",
"full_like",
"group",
Expand Down
28 changes: 3 additions & 25 deletions src/zarr/api/asynchronous.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import numpy.typing as npt
from typing_extensions import deprecated

from zarr.core.array import Array, AsyncArray, create_array, get_array_metadata
from zarr.core.array import Array, AsyncArray, create_array, from_array, get_array_metadata
from zarr.core.array_spec import ArrayConfig, ArrayConfigLike
from zarr.core.buffer import NDArrayLike
from zarr.core.common import (
Expand All @@ -21,10 +21,8 @@
_default_zarr_format,
_warn_order_kwarg,
_warn_write_empty_chunks_kwarg,
concurrent_map,
parse_dtype,
)
from zarr.core.config import config
from zarr.core.group import AsyncGroup, ConsolidatedMetadata, GroupMetadata
from zarr.core.metadata import ArrayMetadataDict, ArrayV2Metadata, ArrayV3Metadata
from zarr.core.metadata.v2 import _default_compressor, _default_filters
Expand Down Expand Up @@ -52,6 +50,7 @@
"create_array",
"empty",
"empty_like",
"from_array",
"full",
"full_like",
"group",
Expand Down Expand Up @@ -547,28 +546,7 @@ async def array(
"""

if isinstance(data, Array):
# fill missing arguments with metadata of data Array
kwargs.setdefault("dtype", data.dtype)
kwargs.setdefault("attributes", data.attrs)
kwargs.setdefault("chunks", data.chunks)
kwargs.setdefault("fill_value", data.fill_value)

new_array = await create(data.shape, **kwargs)

async def _copy_chunk(chunk_coords: ChunkCoords | slice, _data: Array) -> None:
arr = await _data._async_array.getitem(chunk_coords)
await new_array.setitem(chunk_coords, arr)

if new_array.chunks == data.chunks:
# Stream data from the source array to the new array
await concurrent_map(
[(region, data) for region in data._iter_chunk_regions()],
_copy_chunk,
config.get("async.concurrency"),
)
else:
await _copy_chunk(slice(None), data)
return new_array
return await from_array(data, **kwargs)

# ensure data is array-like
if not hasattr(data, "shape") or not hasattr(data, "dtype"):
Expand Down
133 changes: 133 additions & 0 deletions src/zarr/api/synchronous.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
"create_array",
"empty",
"empty_like",
"from_array",
"full",
"full_like",
"group",
Expand Down Expand Up @@ -893,6 +894,138 @@ def create_array(
)


def from_array(
data: Array,
store: str | StoreLike,
*,
name: str | None = None,
chunks: ChunkCoords | Literal["auto"] = "auto",
shards: ShardsLike | None = None,
filters: FiltersLike = "auto",
compressors: CompressorsLike = "auto",
serializer: SerializerLike = "auto",
fill_value: Any | None = None,
order: MemoryOrder | None = None,
zarr_format: ZarrFormat | None = 3,
attributes: dict[str, JSON] | None = None,
chunk_key_encoding: ChunkKeyEncoding | ChunkKeyEncodingLike | None = None,
dimension_names: Iterable[str] | None = None,
storage_options: dict[str, Any] | None = None,
overwrite: bool = False,
config: ArrayConfig | ArrayConfigLike | None = None,
) -> Array:
"""Create an array from an existing array.

Parameters
----------
data : Array
The array to copy.
store : str or Store
Store or path to directory in file system or name of zip file for the new array.
name : str or None, optional
The name of the array within the store. If ``name`` is ``None``, the array will be located
at the root of the store.
chunks : ChunkCoords, optional
Chunk shape of the array.
If not specified, defaults to the chunk shape of the data array.
shards : ChunkCoords, optional
Shard shape of the array. The default value of ``None`` results in no sharding at all.
filters : Iterable[Codec], optional
Iterable of filters to apply to each chunk of the array, in order, before serializing that
chunk to bytes.

For Zarr format 3, a "filter" is a codec that takes an array and returns an array,
and these values must be instances of ``ArrayArrayCodec``, or dict representations
of ``ArrayArrayCodec``.

For Zarr format 2, a "filter" can be any numcodecs codec; you should ensure that the
the order if your filters is consistent with the behavior of each filter.

If no ``filters`` are provided, defaults to the filters of the data array.
compressors : Iterable[Codec], optional
List of compressors to apply to the array. Compressors are applied in order, and after any
filters are applied (if any are specified) and the data is serialized into bytes.

For Zarr format 3, a "compressor" is a codec that takes a bytestream, and
returns another bytestream. Multiple compressors my be provided for Zarr format 3.

For Zarr format 2, a "compressor" can be any numcodecs codec. Only a single compressor may
be provided for Zarr format 2.

If no ``compressors`` are provided, defaults to the compressors of the data array.
serializer : dict[str, JSON] | ArrayBytesCodec, optional
Array-to-bytes codec to use for encoding the array data.
Zarr format 3 only. Zarr format 2 arrays use implicit array-to-bytes conversion.

If no ``serializer`` is provided, defaults to the serializer of the input array.
fill_value : Any, optional
Fill value for the array.
If not specified, defaults to the fill value of the data array.
order : {"C", "F"}, optional
The memory of the array (default is "C").
For Zarr format 2, this parameter sets the memory order of the array.
For Zarr format 3, this parameter is deprecated, because memory order
is a runtime parameter for Zarr format 3 arrays. The recommended way to specify the memory
order for Zarr format 3 arrays is via the ``config`` parameter, e.g. ``{'config': 'C'}``.
If not specified, defaults to the memory order of the data array.
zarr_format : {2, 3}, optional
The zarr format to use when saving.
If not specified, defaults to the zarr format of the data array.
attributes : dict, optional
Attributes for the array.
If not specified, defaults to the attributes of the data array.
chunk_key_encoding : ChunkKeyEncoding, optional
A specification of how the chunk keys are represented in storage.
For Zarr format 3, the default is ``{"name": "default", "separator": "/"}}``.
For Zarr format 2, the default is ``{"name": "v2", "separator": "."}}``.
If not specified and the data array has the same zarr format as the target array,
the chunk key encoding of the data array is used.
dimension_names : Iterable[str], optional
The names of the dimensions (default is None).
Zarr format 3 only. Zarr format 2 arrays should not use this parameter.
If not specified, defaults to the dimension names of the data array.
storage_options : dict, optional
If using an fsspec URL to create the store, these will be passed to the backend implementation.
Ignored otherwise.
overwrite : bool, default False
Whether to overwrite an array with the same name in the store, if one exists.
config : ArrayConfig or ArrayConfigLike, optional
Runtime configuration for the array.

Returns
-------
AsyncArray
The array.

Examples
--------
#TODO
"""
return Array(
sync(
zarr.core.array.from_array(
data,
store,
name=name,
chunks=chunks,
shards=shards,
filters=filters,
compressors=compressors,
serializer=serializer,
fill_value=fill_value,
order=order,
zarr_format=zarr_format,
attributes=attributes,
chunk_key_encoding=chunk_key_encoding,
dimension_names=dimension_names,
storage_options=storage_options,
overwrite=overwrite,
config=config,
)
)
)


# TODO: add type annotations for kwargs
def empty(shape: ChunkCoords, **kwargs: Any) -> Array:
"""Create an empty array.
Expand Down
Loading
Loading