Merge branch 'main' into docs/sharding

Author: normanrz

docs/api/index.rst

@@ -1,4 +1,4 @@
-API Reference
+API reference
 =============
 
 .. toctree::

docs/conf.py

@@ -104,6 +104,10 @@ def skip_submodules(
      "spec/v3": "https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html",
      "license": "https://github.com/zarr-developers/zarr-python/blob/main/LICENSE.txt",
      "tutorial": "user-guide",
+     "getting-started": "quickstart",
+     "release": "developers/release.html",
+     "roadmap": "developers/roadmap.html",
+     "installation": "user-guide/installation.html",
 }
 
 # The language for content autogenerated by Sphinx. Refer to documentation

docs/contributing.rst renamed to docs/developers/contributing.rst

@@ -1,5 +1,7 @@
-Contributing
-============
+.. _dev-guide-contributing:
+
+Contributing to Zarr
+====================
 
 Zarr is a community maintained project. We welcome contributions in the form of bug
 reports, bug fixes, documentation, enhancement proposals and more. This page provides
@@ -46,8 +48,7 @@ a bug report:
    interpreter can be obtained by running a Python interactive session, e.g.::
 
     $ python
-    Python 3.6.1 (default, Mar 22 2017, 06:17:05)
-    [GCC 6.3.0 20170321] on linux
+      Python 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 15:57:01) [Clang 17.0.6 ] on darwin
 
 Enhancement proposals
 ---------------------
@@ -73,7 +74,8 @@ The Zarr source code is hosted on GitHub at the following location:
 * `https://github.com/zarr-developers/zarr-python <https://github.com/zarr-developers/zarr-python>`_
 
 You will need your own fork to work on the code. Go to the link above and hit
-the "Fork" button. Then clone your fork to your local machine::
+the `"Fork" <https://github.com/zarr-developers/zarr-python/fork>`_ button.
+Then clone your fork to your local machine::
 
     $ git clone [email protected]:your-user-name/zarr-python.git
     $ cd zarr-python
@@ -82,21 +84,21 @@ the "Fork" button. Then clone your fork to your local machine::
 Creating a development environment
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To work with the Zarr source code, it is recommended to set up a Python virtual
-environment and install all Zarr dependencies using the same versions as are used by
-the core developers and continuous integration services. Assuming you have a Python
-3 interpreter already installed, and you have cloned the Zarr source code and your
-current working directory is the root of the repository, you can do something like
-the following::
+To work with the Zarr source code, it is recommended to use
+`hatch <https://hatch.pypa.io/latest/index.html>`_ to create and manage development
+environments. Hatch will automatically install all Zarr dependencies using the same
+versions as are used by the core developers and continuous integration services.
+Assuming you have a Python 3 interpreter already installed, and you have cloned the
+Zarr source code and your current working directory is the root of the repository,
+you can do something like the following::
 
-    $ mkdir -p ~/pyenv/zarr-dev
-    $ python -m venv ~/pyenv/zarr-dev
-    $ source ~/pyenv/zarr-dev/bin/activate
-    $ pip install -e .[test,docs]
+    $ pip install hatch
+    $ hatch env show  # list all available environments
 
-To verify that your development environment is working, you can run the unit tests::
+To verify that your development environment is working, you can run the unit tests
+for one of the test environments, e.g.::
 
-    $ python -m pytest -v tests
+    $ hatch env run --env test.py3.12-2.1-optional run
 
 Creating a branch
 ~~~~~~~~~~~~~~~~~
@@ -109,9 +111,7 @@ new, separate branch for each piece of work you want to do. E.g.::
 
     git checkout main
     git fetch upstream
-    git rebase upstream/main
-    git push
-    git checkout -b shiny-new-feature
+    git checkout -b shiny-new-feature upstream/main
     git push -u origin shiny-new-feature
 
 This changes your working directory to the 'shiny-new-feature' branch. Keep any changes in
@@ -129,54 +129,27 @@ merge conflicts, these need to be resolved before submitting a pull request.
 Alternatively, you can merge the changes in from upstream/main instead of rebasing,
 which can be simpler::
 
-    git fetch upstream
-    git merge upstream/main
+    git pull upstream main
 
 Again, any conflicts need to be resolved before submitting a pull request.
 
 Running the test suite
 ~~~~~~~~~~~~~~~~~~~~~~
 
-Zarr includes a suite of unit tests, as well as doctests included in
-function and class docstrings and in the tutorial and storage
-spec. The simplest way to run the unit tests is to activate your
-development environment (see `creating a development environment`_ above)
-and invoke::
-
-    $ python -m pytest -v zarr
-
-Some tests require optional dependencies to be installed, otherwise
-the tests will be skipped. To install all optional dependencies, run::
-
-    $ pip install pytest-doctestplus
-
-To also run the doctests within docstrings (requires optional
-dependencies to be installed), run::
-
-    $ python -m pytest -v --doctest-plus zarr
-
-To run the doctests within the tutorial and storage spec (requires
-optional dependencies to be installed), run::
-
-    $ python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst docs/spec/v2.rst
-
-Note that some tests also require storage services to be running
-locally. To run the Azure Blob Service storage tests, run an Azure
-storage emulator (e.g., azurite) and set the environment variable
-``ZARR_TEST_ABS=1``. If you're using Docker to run azurite, start the service with::
+Zarr includes a suite of unit tests. The simplest way to run the unit tests
+is to activate your development environment
+(see `creating a development environment`_ above) and invoke::
 
-    docker run --rm -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite-blob --loose --blobHost 0.0.0.0
-
-To run the Mongo DB storage tests, run a Mongo
-server locally and set the environment variable ``ZARR_TEST_MONGO=1``.
-To run the Redis storage tests, run a Redis server locally on port
-6379 and set the environment variable ``ZARR_TEST_REDIS=1``.
+    $ hatch env run --env test.py3.12-2.1-optional run
 
 All tests are automatically run via GitHub Actions for every pull
 request and must pass before code can be accepted. Test coverage is
-also collected automatically via the Codecov service, and total
-coverage over all builds must be 100% (although individual builds
-may be lower due to Python 2/3 or other differences).
+also collected automatically via the Codecov service.
+
+.. note::
+    Previous versions of Zarr-Python made extensive use of doctests. These tests were
+    not maintained during the 3.0 refactor but may be brought back in the future.
+    See :issue:`2614` for more details.
 
 Code standards - using pre-commit
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -205,15 +178,17 @@ If you would like to skip the failing checks and push the code for further discu
 the ``--no-verify`` option with ``git commit``.
 
 
-
 Test coverage
 ~~~~~~~~~~~~~
 
-Zarr maintains 100% test coverage under the latest Python stable release (currently
-Python 3.8). Both unit tests and docstring doctests are included when computing
-coverage. Running::
+.. note::
+    Test coverage for Zarr-Python 3 is currently not at 100%. This is a known issue and help
+    is welcome to bring test coverage back to 100%. See :issue:`2613` for more details.
+
+Zarr strives to maintain 100% test coverage under the latest Python stable release
+Both unit tests and docstring doctests are included when computing coverage. Running::
 
-    $ python -m pytest -v --cov=zarr --cov-config=pyproject.toml zarr
+    $ hatch env run --env test.py3.12-2.1-optional run-coverage
 
 will automatically run the test suite with coverage and produce a coverage report.
 This should be 100% before code can be accepted into the main code base.
@@ -229,28 +204,28 @@ Docstrings for user-facing classes and functions should follow the
 `numpydoc
 <https://numpydoc.readthedocs.io/en/stable/format.html#docstring-standard>`_
 standard, including sections for Parameters and Examples. All examples
-should run and pass as doctests under Python 3.8. To run doctests,
-activate your development environment, install optional requirements,
-and run::
-
-    $ python -m pytest -v --doctest-plus tests
+should run and pass as doctests under Python 3.11.
 
 Zarr uses Sphinx for documentation, hosted on readthedocs.org. Documentation is
 written in the RestructuredText markup language (.rst files) in the ``docs`` folder.
 The documentation consists both of prose and API documentation. All user-facing classes
-and functions should be included in the API documentation, under the ``docs/api``
-folder. Any new features or important usage information should be included in the
-tutorial (``docs/tutorial.rst``). Any changes should also be included in the release
-notes (``docs/release.rst``).
+and functions are included in the API documentation, under the ``docs/api`` folder
+using the `autodoc <https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html>`_
+extension to sphinx. Any new features or important usage information should be included in the
+user-guide (``docs/user-guide``). Any changes should also be included in the release
+notes (``docs/developers/release.rst``).
 
 The documentation can be built locally by running::
 
-    $ cd docs
-    $ make clean; make html
-    $ open _build/html/index.html
+    $ hatch --env docs run build
 
 The resulting built documentation will be available in the ``docs/_build/html`` folder.
 
+Hatch can also be used to serve continuously updating version of the documentation
+during development at `http://0.0.0.0:8000/ <http://0.0.0.0:8000/>`_. This can be done by running::
+
+    $ hatch --env docs run serve
+
 Development best practices, policies and procedures
 ---------------------------------------------------
 
@@ -329,14 +304,7 @@ implements storage spec version 3, then the next library release should have ver
 number 3.0.0. Note however that the major version number of the Zarr library may not
 always correspond to the spec version number. For example, Zarr versions 2.x, 3.x, and
 4.x might all implement the same version of the storage spec and thus maintain data
-format compatibility, although they will not maintain API compatibility. The version number
-of the storage specification that is currently implemented is stored under the
-``zarr.meta.ZARR_FORMAT`` variable.
-
-Note that the Zarr test suite includes a data fixture and tests to try and ensure that
-data format compatibility is not accidentally broken. See the
-:func:`test_format_compatibility` function in the :mod:`tests.test_storage` module
-for details.
+format compatibility, although they will not maintain API compatibility.
 
 When to make a release
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -358,7 +326,7 @@ Release procedure
 
 .. note::
 
-   Most of the release process is now handled by github workflow which should
+   Most of the release process is now handled by GitHub workflow which should
    automatically push a release to PyPI if a tag is pushed.
 
 Before releasing, make sure that all pull requests which will be

docs/developers/index.rst

@@ -0,0 +1,10 @@
+
+Developer's Guide
+-----------------
+
+.. toctree::
+    :maxdepth: 1
+
+    contributing
+    release
+    roadmap

docs/release.rst renamed to docs/developers/release.rst

@@ -8,59 +8,60 @@ Zarr-Python
     :maxdepth: 1
     :hidden:
 
-    getting_started
-    about
+    quickstart
     user-guide/index
     api/index
-    release
-    contributing
-    roadmap
+    developers/index
+    developers/release
+    about
 
 **Version**: |version|
 
-**Download documentation**: `PDF/Zipped HTML <https://readthedocs.org/projects/zarr/downloads/>`_
-
 **Useful links**:
-`Installation <installation.html>`_ |
 `Source Repository <https://github.com/zarr-developers/zarr-python>`_ |
 `Issue Tracker <https://github.com/zarr-developers/zarr-python/issues>`_ |
 `Zulip Chat <https://ossci.zulipchat.com/>`_ |
 `Zarr specifications <https://zarr-specs.readthedocs.io>`_
 
-Zarr is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification.
+Zarr-Python is a Python library for reading and writing Zarr groups and arrays. Highlights include:
+
+* Specification support for both Zarr v2 and v3.
+* Create and read from N-dimensional arrays using NumPy-like semantics.
+* Flexible storage enables reading and writing from local, cloud and in-memory stores.
+* High performance: Enables fast I/O with support for asynchronous I/O and multi-threading.
+* Extensible: Customizable with user-defined codecs and stores.
 
 .. grid:: 2
 
     .. grid-item-card::
         :img-top: _static/index_getting_started.svg
 
-        Getting Started
-        ^^^^^^^^^^^^^^^
+        Quick Start
+        ^^^^^^^^^^^
 
-        New to Zarr? Check out the getting started guide. It contains an
+        New to Zarr? Check out the quick start guide. It contains a brief
         introduction to Zarr's main concepts and links to additional tutorials.
 
         +++
 
-        .. button-ref:: getting_started
+        .. button-ref:: quickstart
             :expand:
             :color: dark
             :click-parent:
 
-            To the getting started guide
+            To the Quick Start
 
     .. grid-item-card::
         :img-top: _static/index_user_guide.svg
 
-       Guide
-       ^^^^^
+        Guide
+        ^^^^^
 
-       A detailed guide for how to use Zarr-Python.
+        A detailed guide for how to use Zarr-Python.
 
         +++
 
-        .. button-ref:: user-guide
-            :ref-type: ref
+        .. button-ref:: user-guide/index
             :expand:
             :color: dark
             :click-parent:
@@ -85,21 +86,28 @@ Zarr is a file storage format for chunked, compressed, N-dimensional arrays base
             :color: dark
             :click-parent:
 
-            To the api reference guide
+            To the API reference guide
 
     .. grid-item-card::
         :img-top: _static/index_contribute.svg
 
         Contributor's Guide
         ^^^^^^^^^^^^^^^^^^^
 
-        Want to contribute to Zarr? We welcome contributions in the form of bug reports, bug fixes, documentation, enhancement proposals and more. The contributing guidelines will guide you through the process of improving Zarr.
+        Want to contribute to Zarr? We welcome contributions in the form of bug reports,
+        bug fixes, documentation, enhancement proposals and more. The contributing guidelines
+        will guide you through the process of improving Zarr.
 
         +++
 
-        .. button-ref:: contributing
+        .. button-ref:: developers/contributing
             :expand:
             :color: dark
             :click-parent:
 
             To the contributor's guide
+
+
+**Download documentation**: `PDF/Zipped HTML <https://readthedocs.org/projects/zarr/downloads/>`_
+
+.. _NumCodecs: https://numcodecs.readthedocs.io