Merge branch 'main' of https://github.com/zarr-developers/zarr-python into docs/3.0-dev-guide

Author: jhamman

.github/workflows/test.yml

@@ -94,13 +94,38 @@ jobs:
       run: |
         hatch env run --env ${{ matrix.dependency-set }} run
 
+  doctests:
+    name: doctests
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        fetch-depth: 0 # required for hatch version discovery, which is needed for numcodecs.zarr3
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.13'
+        cache: 'pip'
+    - name: Install Hatch
+      run: |
+        python -m pip install --upgrade pip
+        pip install hatch
+    - name: Set Up Hatch Env
+      run: |
+        hatch env create doctest
+        hatch env run -e doctest list-env
+    - name: Run Tests
+      run: |
+        hatch env run --env doctest run
+
   test-complete:
     name: Test complete
 
     needs:
       [
         test,
         test-upstream-and-min-deps,
+        doctests
       ]
     if: always()
     runs-on: ubuntu-latest
@@ -111,4 +136,4 @@ jobs:
           contains(needs.*.result, 'cancelled')
         run: exit 1
       - name: Success
-        run: echo Success!
+        run: echo Success!

.gitignore

@@ -54,6 +54,7 @@ docs/_build/
 docs/_autoapi
 docs/data
 data
+data.zip
 
 # PyBuilder
 target/

docs/about.rst

@@ -0,0 +1,24 @@
+About
+=====
+
+Zarr is a format for the storage of chunked, compressed, N-dimensional arrays
+inspired by `HDF5 <https://www.hdfgroup.org/HDF5/>`_, `h5py
+<https://www.h5py.org/>`_ and `bcolz <https://bcolz.readthedocs.io/>`_.
+
+These documents describe the Zarr-Python implementation. More information
+about the Zarr format can be found on the `main website <https://zarr.dev>`_.
+
+Projects using Zarr
+-------------------
+
+If you are using Zarr-Python, we would `love to hear about it
+<https://github.com/zarr-developers/community/issues/19>`_.
+
+Funding
+-------
+The project is fiscally sponsored by `NumFOCUS <https://numfocus.org/>`_, a US
+501(c)(3) public charity, and development is supported by the
+`MRC Centre for Genomics and Global Health <https://www.cggh.org>`_
+and the `Chan Zuckerberg Initiative <https://chanzuckerberg.com/>`_.
+
+.. _NumCodecs: https://numcodecs.readthedocs.io/

docs/conf.py

@@ -17,6 +17,7 @@
 import sys
 from typing import Any
 
+import sphinx
 import sphinx.application
 
 from importlib.metadata import version as get_version
@@ -48,8 +49,6 @@
     "sphinx_copybutton",
     "sphinx_design",
     'sphinx_reredirects',
-    "IPython.sphinxext.ipython_directive",
-    "IPython.sphinxext.ipython_console_highlighting",
 ]
 
 issues_github_path = "zarr-developers/zarr-python"
@@ -62,6 +61,20 @@
 autoapi_keep_files = True
 autoapi_options = [ 'members', 'undoc-members', 'show-inheritance', 'show-module-summary', 'imported-members', ]
 
+def skip_submodules(
+        app: sphinx.application.Sphinx,
+        what: str,
+        name: str,
+        obj: object,
+        skip: bool,
+        options: dict[str, Any]
+      ) -> bool:
+    # Skip documenting zarr.codecs submodules
+    # codecs are documented in the main zarr.codecs namespace
+    if what == "module" and name.startswith("zarr.codecs."):
+        skip = True
+    return skip
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 
@@ -91,6 +104,7 @@
      "spec/v3": "https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html",
      "license": "https://github.com/zarr-developers/zarr-python/blob/main/LICENSE.txt",
      "tutorial": "user-guide",
+     "getting-started": "quickstart",
      "release": "developers/release.html",
      "roadmap": "developers/roadmap.html",
 }
@@ -184,6 +198,7 @@
 
 def setup(app: sphinx.application.Sphinx) -> None:
     app.add_css_file("custom.css")
+    app.connect("autoapi-skip-member", skip_submodules)
 
 
 # The name of an image file (relative to this directory) to use as a favicon of

docs/developers/release.rst

@@ -1906,7 +1906,7 @@ Enhancements
 
 * **Advanced indexing**. The ``Array`` class has several new methods and
   properties that enable a selection of items in an array to be retrieved or
-  updated. See the :ref:`tutorial_indexing` tutorial section for more
+  updated. See the :ref:`user-guide-indexing` tutorial section for more
   information. There is also a `notebook
   <https://github.com/zarr-developers/zarr-python/blob/main/notebooks/advanced_indexing.ipynb>`_
   with extended examples and performance benchmarks. :issue:`78`, :issue:`89`,
@@ -1919,15 +1919,15 @@ Enhancements
   compressor codecs for Zstd and LZ4. This change is backwards-compatible with
   existing code, as all codec classes defined by Numcodecs are imported into the
   :mod:`zarr.codecs` namespace. However, it is recommended to import codecs from
-  the new package, see the tutorial sections on :ref:`tutorial_compress` and
-  :ref:`tutorial_filters` for examples. With contributions by
+  the new package, see the tutorial sections on :ref:`user-guide-compress` and
+  :ref:`user-guide-filters` for examples. With contributions by
   :user:`John Kirkham <jakirkham>`; :issue:`74`, :issue:`102`, :issue:`120`,
   :issue:`123`, :issue:`139`.
 
 * **New storage class for DBM-style databases**. The
   :class:`zarr.storage.DBMStore` class enables any DBM-style database such as gdbm,
   ndbm or Berkeley DB, to be used as the backing store for an array or group. See the
-  tutorial section on :ref:`tutorial_storage` for some examples. :issue:`133`,
+  tutorial section on :ref:`user-guide-storage` for some examples. :issue:`133`,
   :issue:`186`.
 
 * **New storage class for LMDB databases**. The :class:`zarr.storage.LMDBStore` class
@@ -1943,7 +1943,7 @@ Enhancements
   :func:`zarr.hierarchy.Group.tree` method which enables a tree representation of
   a group hierarchy to be printed. Also provides an interactive tree
   representation when used within a Jupyter notebook. See the
-  :ref:`tutorial_diagnostics` tutorial section for examples. By
+  :ref:`user-guide-diagnostics` tutorial section for examples. By
   :user:`John Kirkham <jakirkham>`; :issue:`82`, :issue:`140`, :issue:`184`.
 
 * **Visitor API**. The ``Group`` class now implements the h5py visitor API, see
@@ -1963,7 +1963,7 @@ Enhancements
   store. The functions :func:`zarr.convenience.save` and
   :func:`zarr.convenience.load` are also available and provide a convenient way to
   save an entire NumPy array to disk and load back into memory later. See the
-  tutorial section :ref:`tutorial_persist` for examples. :issue:`104`,
+  tutorial section :ref:`user-guide-persist` for examples. :issue:`104`,
   :issue:`105`, :issue:`141`, :issue:`181`.
 
 * **IPython completions**. The ``Group`` class now implements ``__dir__()`` and
@@ -1973,15 +1973,15 @@ Enhancements
 * **New info property; changes to __repr__**. The ``Group`` and
   ``Array`` classes have a new ``info`` property which can be used to print
   diagnostic information, including compression ratio where available. See the
-  tutorial section on :ref:`tutorial_diagnostics` for examples. The string
+  tutorial section on :ref:`user-guide-diagnostics` for examples. The string
   representation (``__repr__``) of these classes has been simplified to ensure
   it is cheap and quick to compute in all circumstances. :issue:`83`,
   :issue:`115`, :issue:`132`, :issue:`148`.
 
 * **Chunk options**. When creating an array, ``chunks=False`` can be specified,
   which will result in an array with a single chunk only. Alternatively,
   ``chunks=True`` will trigger an automatic chunk shape guess. See
-  :ref:`tutorial_chunks` for more on the ``chunks`` parameter. :issue:`106`,
+  :ref:`user-guide-chunks` for more on the ``chunks`` parameter. :issue:`106`,
   :issue:`107`, :issue:`183`.
 
 * **Zero-dimensional arrays** and are now supported; by
@@ -2006,7 +2006,7 @@ Enhancements
   creating an array with ``dtype=object`` was possible but could under certain
   circumstances lead to unexpected errors and/or segmentation faults. To make it easier
   to properly configure an object array, a new ``object_codec`` parameter has been
-  added to array creation functions. See the tutorial section on :ref:`tutorial_objects`
+  added to array creation functions. See the tutorial section on :ref:`user-guide-objects`
   for more information and examples. Also, runtime checks have been added in both Zarr
   and Numcodecs so that segmentation faults are no longer possible, even with a badly
   configured array. This API change is backwards compatible and previous code that created
@@ -2062,16 +2062,16 @@ Documentation
   with any of the material as previously implemented, and so the changes have been made
   in-place in the document without incrementing the document version number. See the
   section on changes in the specification document for more information.
-* A new :ref:`tutorial_indexing` section has been added to the tutorial.
-* A new :ref:`tutorial_strings` section has been added to the tutorial
+* A new :ref:`user-guide-indexing` section has been added to the tutorial.
+* A new :ref:`user-guide-strings` section has been added to the tutorial
   (:issue:`135`, :issue:`175`).
-* The :ref:`tutorial_chunks` tutorial section has been reorganised and updated.
-* The :ref:`tutorial_persist` and :ref:`tutorial_storage` tutorial sections have
+* The :ref:`user-guide-chunks` tutorial section has been reorganised and updated.
+* The :ref:`user-guide-persist` and :ref:`user-guide-storage` tutorial sections have
   been updated with new examples (:issue:`100`, :issue:`101`, :issue:`103`).
-* A new tutorial section on :ref:`tutorial_pickle` has been added (:issue:`91`).
-* A new tutorial section on :ref:`tutorial_datetime` has been added.
-* A new tutorial section on :ref:`tutorial_diagnostics` has been added.
-* The tutorial sections on :ref:`tutorial_sync` and :ref:`tutorial_tips_blosc` have been
+* A new tutorial section on :ref:`user-guide-pickle` has been added (:issue:`91`).
+* A new tutorial section on :ref:`user-guide-datetime` has been added.
+* A new tutorial section on :ref:`user-guide-diagnostics` has been added.
+* The tutorial sections on :ref:`user-guide-sync` and :ref:`user-guide-tips-blosc` have been
   updated to provide information about how to avoid program hangs when using the Blosc
   compressor with multiple processes (:issue:`199`, :issue:`201`).
 
@@ -2177,14 +2177,14 @@ Hierarchies
 ~~~~~~~~~~~
 
 Support has been added for organizing arrays into hierarchies via groups. See
-the tutorial section on :ref:`tutorial_groups` and the :mod:`zarr.hierarchy`
+the tutorial section on :ref:`user-guide-groups` and the :mod:`zarr.hierarchy`
 API docs for more information.
 
 Filters
 ~~~~~~~
 
 Support has been added for configuring filters to preprocess chunk data prior
-to compression. See the tutorial section on :ref:`tutorial_filters` and the
+to compression. See the tutorial section on :ref:`user-guide-filters` and the
 :mod:`zarr.codecs` API docs for more information.
 
 Other changes
@@ -2210,7 +2210,7 @@ Thanks to :user:`Matthew Rocklin <mrocklin>`, :user:`Stephan Hoyer <shoyer>` and
 
 * The bundled Blosc library has been upgraded to version 1.10.0. The 'zstd'
   internal compression library is now available within Blosc. See the tutorial
-  section on :ref:`tutorial_compress` for an example.
+  section on :ref:`user-guide-compress` for an example.
 * When using the Blosc compressor, the default internal compression library
   is now 'lz4'.
 * The default number of internal threads for the Blosc compressor has been
@@ -2236,8 +2236,8 @@ The main motivation for re-organizing the code was to create an
 abstraction layer between the core array logic and data storage (:issue:`21`).
 In this release, any
 object that implements the ``MutableMapping`` interface can be used as
-an array store. See the tutorial sections on :ref:`tutorial_persist`
-and :ref:`tutorial_storage`, the ``spec_v1``, and the
+an array store. See the tutorial sections on :ref:`user-guide-persist`
+and :ref:`user-guide-storage`, the ``spec_v1``, and the
 :mod:`zarr.storage` module documentation for more information.
 
 Please note also that the file organization and file name conventions
@@ -2256,8 +2256,8 @@ chunks. This release still bundles the c-blosc library and uses Blosc
 as the default compressor, however other compressors including zlib,
 BZ2 and LZMA are also now supported via the Python standard
 library. New compressors can also be dynamically registered for use
-with Zarr. See the tutorial sections on :ref:`tutorial_compress` and
-:ref:`tutorial_tips_blosc`, the ``spec_v1``, and the
+with Zarr. See the tutorial sections on :ref:`user-guide-compress` and
+:ref:`user-guide-tips-blosc`, the ``spec_v1``, and the
 :mod:`zarr.compressors` module documentation for more information.
 
 Synchronization
@@ -2266,7 +2266,7 @@ Synchronization
 The synchronization code has also been refactored to create a layer of
 abstraction, enabling Zarr arrays to be used in parallel computations
 with a number of alternative synchronization methods. For more
-information see the tutorial section on :ref:`tutorial_sync` and the
+information see the tutorial section on :ref:`user-guide-sync` and the
 :mod:`zarr.sync` module documentation.
 
 Changes to the Blosc extension
@@ -2288,7 +2288,7 @@ is running within a single-threaded or multi-threaded program and
 adapts its internal behaviour accordingly (:issue:`27`). There is no need for
 the user to make any API calls to switch Blosc between contextual and
 non-contextual (global lock) mode. See also the tutorial section on
-:ref:`tutorial_tips_blosc`.
+:ref:`user-guide-tips-blosc`.
 
 Other changes
 ~~~~~~~~~~~~~
@@ -2302,7 +2302,7 @@ option present in the previous release, and this has been removed.
 The memory layout within chunks can now be set as either "C"
 (row-major) or "F" (column-major), which can help to provide better
 compression for some data (:issue:`7`). See the tutorial
-section on :ref:`tutorial_chunks_order` for more information.
+section on :ref:`user-guide-chunks-order` for more information.
 
 A bug has been fixed within the ``__getitem__`` and ``__setitem__``
 machinery for slicing arrays, to properly handle getting and setting