CHANGELOG.md

HDMF-ZARR Changelog

Upcoming

Added

• Added ZarrIO.generate_dataset_html method to generate rich HTML representations of Zarr arrays for display in Jupyter notebooks. @rly #306
• Added S3 streaming tutorial with consolidated metadata guidance. @rly @oruebel #308

Fixed

• Fixed zarr array .info property to display compression data ("No. bytes stored" and "Storage ratio") when using consolidated metadata stores by patching ConsolidatedMetadataStore.getsize() to query the underlying chunk store. @rly #305
• Fixed bug where the cached spec was not included in consolidated metadata when exporting using ZarrIO. @rly #315
• Fixed bug where consolidated metadata was always written by ZarrIO. @rly #315
• Fixed bug where the specifications group and its contents were read during ZarrIO.read_builder. @rly #322
• Fixed issue with tox.ini configuration. @rly #326
• Fixed ZarrIO.is_remote() returning False for remote stores using consolidated metadata, which caused resolve_ref to mangle HTTPS URLs and fail with PathNotFoundError when reading links. @rly #328

Changed

• Replaced requirements-min.txt with uv pip install --resolution lowest-direct in tox and converted test and docs from optional dependencies to dependency groups (PEP 735), making the project compatible with uv. @h-mayorquin #327
• Updated documentation about how the zarr_dtype attribute is used to specify a compound dtype (structured array) for a dataset. @rly #313
• Dropped testing for Python 3.9. @rly #298
• Added support for Python 3.14. @rly #298
• Updated HDF5 to Zarr conversion tutorial and test data to use a more modern file. @rly #312

0.12.0 (October 8, 2025)

Changed

• Renamed ZarrIO.file property to ZarrIO._file to be consistent with PyNWB validator. @rly #294
• When loading namespaces with ZarrIO, all namespaces are now passed to namespace_catalog.load_namespaces to handle ordering of loading and detection of version conflicts, to be consistent with HDF5IO. @rly #294
• Renamed test module tests/unit/utils.py to tests/unit/helpers/utils.py. #292
• Bumped minimum required version of numcodecs to 0.12.0 to address installation issues with Python 3.9 on MacOS. @rly #299

Added

• Added ZarrIO.load_namespaces_io method to load namespaces from a given Zarr group into a given NamespaceCatalog. @rly #292

0.11.3 (August 8, 2025)

Added

• Cache consolidated metadata in ZarrIO to avoid repeated metadata reads, significantly improving performance for datasets with many references. @rly #290

0.11.2 (July 28, 2025)

Added

• Added ZarrDataIO.get_io_params. @rly #282

Fixed

• Fixed saving of the cached specs in the consolidated metadata. @stephprince #274
• Fixed writing of compound dtype datasets. @mavaylon1 #276
• Fixed pixel mask data shape expansion bug in __list_fill__ method that was incorrectly duplicating compound data type values during export operations. @pauladkisson #280

0.11.1 (April 8, 2025)

Changed

• Added scipy to optional dependencies to be compatible with HDMF 4.0. @rly #261
• Restricted numcodecs dependency to <0.16 due to incompatibility with the latest zarr<3 version. @rly #270

0.11.0 (January 17, 2025)

Changed

• Adopted changelog format conventions: https://keepachangelog.com/en/1.0.0/ . @rly #254
• Made docs site point to latest stable release: https://hdmf-zarr.readthedocs.io/en/stable/ instead of "latest" tag. @rly #254
• Removed GitHub Actions workflow that automatically generated GitHub pre-releases on the "latest" tag on each commit to dev to simplify maintenance. @rly #254
• Clarified docs and updated links to say that only Zarr v2 is currently supported. @rly #257
• Removed ZarrIO.get_zarr_parent_path and ZarrIO.is_zarr_file methods. @rly #256
• Fixed bug in how links and references are stored in the Zarr file. They are now written as relative paths from the Zarr file, using "." to indicate the current file. This is how hdmf-zarr wrote internal links and references pre-0.10.0. @rly #256

0.10.0 (December 18, 2024)

Enhancements

• Added initial refactor of export, supporting references and internal/external links from Zarr to Zarr. This will introduce breaking changes that could lead to existing exported files to be invalid. This update removes '.' as the object default file source. @mavaylon1 #194
• Added support for Pathlib paths. @mavaylon1 #212
• Updated packages used for testing and readthedocs configuration. @mavaylon1, @rly #214
• Add force_overwite parameter for ZarrIO.__init__ to allow overwriting an existing file or directory. @oruebel #229
• Remove allowance of hdmf.Array in __init__ of AbstractZarrTableDataset and ZarrDataset to be compatible with HDMF 4.0. @rly #236
• Remove support for python 3.8 and added testing for Python 3.13. @mavaylon1 #240
• Added NWBZarrIO.read_nwb convenience method to simplify reading an NWB file. @oruebel #226
• Updated optional dependency groups in pyproject.toml and GitHub Actions workflows. @rly, @mavaylon1 #239
• Applied black code formatter. @rly #247

Bug Fixes

• Fix reading of cached specs and caching of specs during export. @rly #232
• Fix hiding of pynwb compatibility errors. @rly 242

0.9.0 (September 16, 2024)

Enhancements

• Added support for appending a dataset of references. @mavaylon1 #203
• NWBZarrIO load_namespaces=True by default. @mavaylon1 #204
• Added test for opening file with consolidated metadata from DANDI. @mavaylon1 #206
• Add dimension labels compatible with xarray. @mavaylon1 #207
• Added link_data --> clear_cache relationship to support repacking zarr nwbfiles: #215

0.8.0 (June 4, 2024)

Bug Fixes

• Fixed bug when opening a file in with mode=r+. The file will open without using the consolidated metadata. @mavaylon1 #182
• Fixed bug on how we access scalar arrays. Added warning filter for Zarr deprecation of NestedDirectoryStore. Fixed bug on how we write a dataset of references. @mavaylon1 #195

0.7.0 (May 2, 2024)

Enhancements

• Added support for python 3.12. @mavaylon1 #172
• Added support for forcing read of files without consolidated metadata using mode=r- in ZarrIO. @oruebel #183

• Updated testing to not install in editable mode and not run coverage by default. @rly #188

Docs

• Removed linkable from the documentation to keep in line with hdmf-schema-language. @mavaylon1 #180

Bug Fixes

• Fixed bug in ZarrIO.__open_file_consolidated that led to remote files being opened without consolidated metadata. @oruebel #184
• Fixed minor bug where ZarrIO.__open_file_consolidated used properties of ZarrIO instead of the provided input parameters. @oruebel #183

0.6.0 (February 21, 2024)

Enhancements

• Enhanced ZarrIO and ZarrDataIO to infer io settings (e.g., chunking and compression) from HDF5 datasets to preserve storage settings on export if possible @oruebel #153
• Updated writing references in compound datasets to same-sized array, rather than iteratively as an array of lists. @sneakers-the-rat #146

Bug Fixes

• Fixed bug when converting HDF5 datasets with unlimited dimensions @oruebel #155
• Fixed bug resolving bytes dtype when exporting from Zarr to Zarr @oruebel #161
• Adjusted gallery tests to not fail on deprecation warnings from pandas. @rly #157
• Fixed bug in pyproject.toml where duplicate versions of numcodec where specified. @oruebel #163

Internal Fixes

• Updated actions in deploy_release.yml workflow to avoid use of deprecated Node.js 16. @oruebel #165

0.5.0 (December 8, 2023)

Enhancements

• Added a new default to consolidate metadata in order more efficeintly traverse storage contents. @mavaylon1 #142
• Fix linking for FSSpec and support passing of storage_options required reading data from S3 #138. @alejoe91 #120
• Updated versioning to hatch-vcs and deprecated setup.py in full transition to pyproject.toml. @mavaylon1 #143

0.4.0 (October 3, 2023)

Enhancements

• Enhanced ZarrIO to resolve object references lazily on read similar to HDMF's HDF5IO backend. @mavaylon1 #120
• Updated storage of references to also save the object_id and source_object_id. While not strictly necessary this information is useful for validation and rigor of references. @oruebel #57

New Features

• Added parallel write support for the ZarrIO for datasets wrapped with GenericDataChunkIterator. @CodyCBakerPhD #118 @oruebel #128

Dependencies

• Updated HDMF and PyNWB version to the most recent release @mavaylon1 #120
• Updated minimum Python version from 3.7 to 3.8 @mavaylon1 #120
• Added threadpoolctl dependency for parallel write. @CodyCBakerPhD #118
• Added support for specifying optional dependencies and add optional dependency for tqdm used in parallel write. @CodyCBakerPhD #118

Bug fixes

• Fixed error in deploy workflow. @mavaylon1 #109
• Fixed build error for ReadtheDocs by degrading numpy for python 3.7 support. @mavaylon1 #115
• Fixed bug in the resolution of references to Containers. Previously references were only resolved to Builders. @mavaylon1 #120

0.3.0 (July 21, 2023)

New Features

• Added support, tests, and docs for using DirectoryStore, TempStore, and NestedDirectoryStore Zarr storage backends with ZarrIO and NWBZarrIO. @oruebel #62

Minor enhancements

• Updated handling of references on read to simplify future integration of file-based Zarr stores (e.g., ZipStore or database stores). @oruebel #62
• Added can_read classmethod to ZarrIO. @bendichter #97

Test suite enhancements

• Modularized unit tests to simplify running tests for multiple Zarr storage backends. @oruebel #62
• Fixed CI testing of minimum and optional installation requirement. @rly #99
• Updated tests to handle upcoming changes to HDMFIO. @rly #102

Docs

• Added developer documentation on how to integrate new storage backends with ZarrIO. @oruebel #62

API Changes

• Removed unused filepath argument from ZarrIO.get_builder_exists_on_disk #62

Bug fixes

• Fixed error in nightly CI. @rly #93

0.2.0 (January 6, 2023)

Bugs

• Updated the storage of links/references to use paths relative to the current Zarr file to avoid breaking links/reference when moving Zarr files @oruebel #46
• Fixed bugs in requirements defined in setup.py @oruebel #46
• Fixed bug regarding Sphinx external links @mavaylon1 #53
• Updated gallery tests to use test_gallery.py and necessary package dependencies @mavaylon1 #53
• Updated dateset used in conversion tutorial, which caused warnings @oruebel #56

Docs

• Added tutorial illustrating how to create a new NWB file with NWBZarrIO @oruebel #46
• Added docs for describing the mapping of HDMF schema to Zarr storage @oruebel #48
• Added docs/gallery/resources for storing local files used by the tutorial galleries @oruebel #61
• Removed dependency on dandi library for data download in the conversion tutorial by storing the NWB files as local resources @oruebel #61

0.1.0 (August 23, 2022)

New features

• Created new optional Zarr-based I/O backend for writing files using Zarr's zarr.store.DirectoryStore backend, including support for iterative write, chunking, compression, simple and compound data types, links, object references, namespace and spec I/O.