Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/release-notes/0.12.0rc3.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
(v0.12.0rc3)=
### 0.12.0rc3 {small}`2025-05-20`

### Bug fixes
#### Bug fixes

- Update zarr v3 bound to >3.0.8 to prevent corrupted data {issue}`zarr-developers/zarr-python#3061` {user}`ilan-gold` ({pr}`1993`)
4 changes: 4 additions & 0 deletions docs/release-notes/0.12.0rc4.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,7 @@
### Performance

- Improve {func}`~anndata.experimental.read_elem_lazy` performance for `h5ad` files by not caching `indptr`. {user}`ilan-gold` ({pr}`2005`)

#### Development

- Bound {mod}`zarr` to `<3.1` until {pr}`1995` is merged to handle the new data type structure. {user}`ilan-gold` ({pr}`2013`)
1 change: 0 additions & 1 deletion docs/release-notes/2013.development.md

This file was deleted.

17 changes: 17 additions & 0 deletions docs/tutorials/zarr-v3.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,23 @@
Users should notice a significant performance improvement, especially for cloud data, but also likely for local data as well.
Here is a quick guide on some of our learnings so far:

## Consolidated Metadata

All `zarr` stores are now consolidated by default when written via {func}`anndata.io.write_zarr` or {meth}`anndata.AnnData.write_zarr`. For more information on this topic, please seee {ref}`the zarr docs <zarr:user-guide-consolidated-metadata>`. Practcally, this changes means that once a store has been written, it should be treated as immutable **unless you remove the consolidated metadata and/or rewrite after the mutating operation** i.e., if you wish to use `anndata.io.write_elem` to add a column to `obs`, a `layer` etc. to an existing store. For example, to mutate an existing store on-disk, you may do:

```python
g = zarr.open_group(orig_path, mode="a", use_consolidated=False)
ad.io.write_elem(
g,
"obs",
obs,
dataset_kwargs=dict(chunks=(250,)),
)
zarr.consolidate_metadata(g.store)
```

In this example, the store was opened unconsolidated (trying to open it as a consolidated store would error out), edited, and then reconsolidated. Alternatively, one could simple delete the file containing the consolidated metadata first at the root, `.zmetadata`.

## Remote data

We now provide the {func}`anndata.experimental.read_lazy` feature for reading as much of the {class}`~anndata.AnnData` object as lazily as possible, using `dask` and {mod}`xarray`.
Expand Down
19 changes: 15 additions & 4 deletions src/anndata/_io/specs/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -360,11 +360,22 @@
dest_type = type(store)

# Normalize k to absolute path
if (isinstance(store, ZarrGroup) and is_zarr_v2()) or (
isinstance(store, h5py.Group) and not PurePosixPath(k).is_absolute()
):
if (
is_zarr_v2_store := (
(is_zarr_store := isinstance(store, ZarrGroup)) and is_zarr_v2()
)
) or (isinstance(store, h5py.Group) and not PurePosixPath(k).is_absolute()):
k = str(PurePosixPath(store.name) / k)

is_consolidated = False
if is_zarr_v2_store:
from zarr.storage import ConsolidatedMetadataStore

is_consolidated = isinstance(store.store, ConsolidatedMetadataStore)
elif is_zarr_store:
is_consolidated = store.metadata.consolidated_metadata is not None

Check warning on line 375 in src/anndata/_io/specs/registry.py

View check run for this annotation

Codecov / codecov/patch

src/anndata/_io/specs/registry.py#L375

Added line #L375 was not covered by tests
if is_consolidated:
msg = "Cannot overwrite/edit a store with consolidated metadata"
raise ValueError(msg)
if k == "/":
if isinstance(store, ZarrGroup) and not is_zarr_v2():
from zarr.core.sync import sync
Expand Down
14 changes: 14 additions & 0 deletions tests/test_readwrite.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import pandas as pd
import pytest
import zarr
import zarr.convenience
from numba.core.errors import NumbaDeprecationWarning
from scipy.sparse import csc_array, csc_matrix, csr_array, csr_matrix

Expand Down Expand Up @@ -966,3 +967,16 @@ def test_read_lazy_import_error(func, tmp_path):
tmp_path if func is ad.experimental.read_lazy else tmp_path / "obs"
)
)


def test_write_elem_consolidated(tmp_path: Path):
ad.AnnData(np.ones((10, 10))).write_zarr(tmp_path)
g = (
zarr.convenience.open_consolidated(tmp_path)
if is_zarr_v2()
else zarr.open(tmp_path)
)
with pytest.raises(
ValueError, match="Cannot overwrite/edit a store with consolidated metadata"
):
ad.io.write_elem(g["obs"], "foo", np.arange(10))