docs: major refactor in preparation for the 3.0 release

Author: jhamman

.gitignore

@@ -52,6 +52,7 @@ coverage.xml
 # Sphinx documentation
 docs/_build/
 docs/_autoapi
+docs/data
 
 # PyBuilder
 target/

data/donotdelete

@@ -48,6 +48,8 @@
     "sphinx_copybutton",
     "sphinx_design",
     'sphinx_reredirects',
+    "IPython.sphinxext.ipython_directive",
+    "IPython.sphinxext.ipython_console_highlighting",
 ]
 
 issues_github_path = "zarr-developers/zarr-python"
@@ -87,6 +89,11 @@
      "spec/v1": 'https://zarr-specs.readthedocs.io/en/latest/v1/v1.0.html',
      "spec/v2": "https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html",
      "spec/v3": "https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html",
+     "tutorial": "user-guide",
+     "contributing": "developers/contributing.html",
+     "release": "developers/release.html",
+     "roadmap": "developers/roadmap.html",
+     "installation": "user-guide/installation.html",
 }
 
 # The language for content autogenerated by Sphinx. Refer to documentation

docs/conf.py

@@ -33,7 +33,7 @@ a bug report:
     # etc.
     ```
 
-2. An explanation of why the current behaviour is wrong/not desired, and what you
+2. An explanation of why the current behavior is wrong/not desired, and what you
    expect instead.
 
 3. Information about the version of Zarr, along with versions of dependencies and the
@@ -46,8 +46,7 @@ a bug report:
    interpreter can be obtained by running a Python interactive session, e.g.::
 
     $ python
-    Python 3.6.1 (default, Mar 22 2017, 06:17:05)
-    [GCC 6.3.0 20170321] on linux
+      Python 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 15:57:01) [Clang 17.0.6 ] on darwin
 
 Enhancement proposals
 ---------------------
@@ -82,21 +81,21 @@ the "Fork" button. Then clone your fork to your local machine::
 Creating a development environment
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To work with the Zarr source code, it is recommended to set up a Python virtual
-environment and install all Zarr dependencies using the same versions as are used by
+To work with the Zarr source code, it is recommended to use `hatch
+<https://hatch.pypa.io/latest/index.html>`_ to create a virtual environment and
+install all Zarr dependencies using the same versions as are used by
 the core developers and continuous integration services. Assuming you have a Python
 3 interpreter already installed, and you have cloned the Zarr source code and your
 current working directory is the root of the repository, you can do something like
 the following::
 
-    $ mkdir -p ~/pyenv/zarr-dev
-    $ python -m venv ~/pyenv/zarr-dev
-    $ source ~/pyenv/zarr-dev/bin/activate
-    $ pip install -e .[test,docs]
+    $ pip install hatch
+    $ hatch env show  # list all available environments
 
-To verify that your development environment is working, you can run the unit tests::
+To verify that your development environment is working, you can run the unit tests
+for one of the test environments, e.g.::
 
-    $ python -m pytest -v tests
+    $ hatch env run --env test.py3.12-2.1-optional run
 
 Creating a branch
 ~~~~~~~~~~~~~~~~~
@@ -143,40 +142,11 @@ spec. The simplest way to run the unit tests is to activate your
 development environment (see `creating a development environment`_ above)
 and invoke::
 
-    $ python -m pytest -v zarr
-
-Some tests require optional dependencies to be installed, otherwise
-the tests will be skipped. To install all optional dependencies, run::
-
-    $ pip install pytest-doctestplus
-
-To also run the doctests within docstrings (requires optional
-dependencies to be installed), run::
-
-    $ python -m pytest -v --doctest-plus zarr
-
-To run the doctests within the tutorial and storage spec (requires
-optional dependencies to be installed), run::
-
-    $ python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst docs/spec/v2.rst
-
-Note that some tests also require storage services to be running
-locally. To run the Azure Blob Service storage tests, run an Azure
-storage emulator (e.g., azurite) and set the environment variable
-``ZARR_TEST_ABS=1``. If you're using Docker to run azurite, start the service with::
-
-    docker run --rm -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite-blob --loose --blobHost 0.0.0.0
-
-To run the Mongo DB storage tests, run a Mongo
-server locally and set the environment variable ``ZARR_TEST_MONGO=1``.
-To run the Redis storage tests, run a Redis server locally on port
-6379 and set the environment variable ``ZARR_TEST_REDIS=1``.
+    $ hatch env run --env test.py3.12-2.1-optional run
 
 All tests are automatically run via GitHub Actions for every pull
 request and must pass before code can be accepted. Test coverage is
-also collected automatically via the Codecov service, and total
-coverage over all builds must be 100% (although individual builds
-may be lower due to Python 2/3 or other differences).
+also collected automatically via the Codecov service.
 
 Code standards - using pre-commit
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -209,11 +179,11 @@ the ``--no-verify`` option with ``git commit``.
 Test coverage
 ~~~~~~~~~~~~~
 
-Zarr maintains 100% test coverage under the latest Python stable release (currently
-Python 3.8). Both unit tests and docstring doctests are included when computing
+Zarr strives to maintain 100% test coverage under the latest Python stable release
+(currently Python 3.12). Both unit tests and docstring doctests are included when computing
 coverage. Running::
 
-    $ python -m pytest -v --cov=zarr --cov-config=pyproject.toml zarr
+    $ hatch env run --env test.py3.12-2.1-optional run-coverage
 
 will automatically run the test suite with coverage and produce a coverage report.
 This should be 100% before code can be accepted into the main code base.
@@ -229,11 +199,7 @@ Docstrings for user-facing classes and functions should follow the
 `numpydoc
 <https://numpydoc.readthedocs.io/en/stable/format.html#docstring-standard>`_
 standard, including sections for Parameters and Examples. All examples
-should run and pass as doctests under Python 3.8. To run doctests,
-activate your development environment, install optional requirements,
-and run::
-
-    $ python -m pytest -v --doctest-plus tests
+should run and pass as doctests under Python 3.10.
 
 Zarr uses Sphinx for documentation, hosted on readthedocs.org. Documentation is
 written in the RestructuredText markup language (.rst files) in the ``docs`` folder.
@@ -245,9 +211,7 @@ notes (``docs/release.rst``).
 
 The documentation can be built locally by running::
 
-    $ cd docs
-    $ make clean; make html
-    $ open _build/html/index.html
+    $ hatch --env docs run build
 
 The resulting built documentation will be available in the ``docs/_build/html`` folder.
 

docs/contributing.rst renamed to docs/developers/contributing.rst

@@ -0,0 +1,8 @@
+
+.. toctree::
+    :maxdepth: 1
+    :hidden:
+
+    release
+    contributing
+    roadmap

docs/developers/index.rst

@@ -8,65 +8,72 @@ Zarr-Python
     :maxdepth: 1
     :hidden:
 
-    getting_started
-    tutorial
-    guide/index
+    quickstart
+    user-guide/index
     api/index
-    release
-    license
-    contributing
-    roadmap
+    developers/contributing
+    developers/release
 
 **Version**: |version|
 
 **Download documentation**: `PDF/Zipped HTML <https://readthedocs.org/projects/zarr/downloads/>`_
 
 **Useful links**:
-`Installation <installation.html>`_ |
 `Source Repository <https://github.com/zarr-developers/zarr-python>`_ |
 `Issue Tracker <https://github.com/zarr-developers/zarr-python/issues>`_ |
 `Zulip Chat <https://ossci.zulipchat.com/>`_ |
 `Zarr specifications <https://zarr-specs.readthedocs.io>`_
 
-Zarr is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification.
+Zarr is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification. Highlights include:
+
+* Create N-dimensional arrays with any NumPy dtype.
+* Chunk arrays along any dimension.
+* Compress and/or filter chunks using any Numcodecs_ codec.
+* Store arrays in memory, on disk, inside a Zip file, on S3, ...
+* Read an array concurrently from multiple threads or processes.
+* Write to an array concurrently from multiple threads or processes.
+* Organize arrays into hierarchies via groups.
+
+Zarr-Python Documentation
+-------------------------
 
 .. grid:: 2
 
     .. grid-item-card::
         :img-top: _static/index_getting_started.svg
 
-        Getting Started
-        ^^^^^^^^^^^^^^^
+        Quick Start
+        ^^^^^^^^^^^
 
-        New to Zarr? Check out the getting started guide. It contains an
+        New to Zarr? Check out the quick start guide. It contains an
         introduction to Zarr's main concepts and links to additional tutorials.
 
         +++
 
-        .. button-ref:: getting_started
+        .. button-ref:: quickstart
             :expand:
             :color: dark
             :click-parent:
 
-            To the getting started guide
+            To the Quick Start
 
     .. grid-item-card::
         :img-top: _static/index_user_guide.svg
 
-        Tutorial
-        ^^^^^^^^
+        Guide
+        ^^^^^
 
-        The tutorial provides working examples of Zarr classes and functions.
+        The user guide provides a detailed guide for how to use Zarr-Python.
 
         +++
 
-        .. button-ref:: tutorial
+        .. button-ref:: user-guide
             :ref-type: ref
             :expand:
             :color: dark
             :click-parent:
 
-            To the Tutorial
+            To the User Guide
 
     .. grid-item-card::
         :img-top: _static/index_api.svg
@@ -98,9 +105,11 @@ Zarr is a file storage format for chunked, compressed, N-dimensional arrays base
 
         +++
 
-        .. button-ref:: contributing
+        .. button-ref:: developers/contributing
             :expand:
             :color: dark
             :click-parent:
 
             To the contributor's guide
+
+.. _Numcodecs: https://numcodecs.readthedocs.io