Merge branch 'main' into polish_README

Author: valeriupredoi

.readthedocs.yaml

@@ -0,0 +1,30 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-lts-latest
+  tools:
+    # DO NOT use mambaforge-*; that is currently sunsetted
+    python: "miniconda-latest"
+  jobs:
+    post_create_environment:
+      - conda run -n ${CONDA_DEFAULT_ENV} pip install . --no-deps
+
+# Declare the requirements required to build your docs
+conda:
+  environment:
+    environment.yml
+
+# Build documentation in the doc directory with Sphinx
+sphinx:
+  configuration: doc/conf.py
+  fail_on_warning: true
+
+# If using Sphinx, optionally build your docs in additional formats such as PDF
+formats:
+  - pdf

conda-linux-64.lock

@@ -165,7 +165,7 @@
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
 # FIXME add a logo
-html_logo = "figures/alpaca-logo.png"
+html_logo = "figures/PyActiveStorage-logo-complete.jpg"
 
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32

doc/conf.py

@@ -1,6 +1,23 @@
 Introduction
 ************
 
-About
-=====
+About PyActiveStorage
+=====================
 
+PyActiveStorage provides a Python client to local or remote “active” storage (aka “computational” storage), that is,
+storage that instead of just receiving reads and returning blocks of data, can receive requests for, and return the
+results of, computations on blocks of data. We have implemented active storage which supports a limited set of reduction
+computations. We have two implementations of active storage with which PyActiveStorage can interact:
+
+* a prototype in DDN InfiniaTM
+* a production ready `“Reductionist” <https://github.com/stackhpc/reductionist-rs>`_  middleware software which can be
+deployed to turn any S3 object store into active storage
+
+Using PyActiveStorage can speed up workflows, especially if the data is remote, or the local area network is congested.
+Early results accessing global fields of 10km resolution data on the CEDA-JASMIN S3 object store show that time-series
+of hemispheric means can be returned to remote users on low-bandwidth networks (and even in
+different hemispheres) in times which are competitive with local users accessing POSIX data across a LAN.
+
+We anticipate this technology will be game-changing for remote access to data, particularly where there is
+insufficient local storage to make copies of data, or those with low-bandwidth networks – although anyone can benefit from
+minimising network contention and reducing the carbon cost of moving data.

doc/introduction.rst

@@ -4,4 +4,61 @@
 Configuration
 *************
 
+Configuring a run with S3 data
+------------------------------
 
+As a standalone application, PyActiveStorage does not need any configuration, data being
+supplied directly to the ``Active`` class; however, some configuration is still needed to be
+able to access an S3 bucket.
+
+For an example, suppose we are trying to access a file ``ch330a.pc19790301-def.nc`` on CEDA-JASMIN's S3 storage,
+for which one needs to supply ``Active`` the ``storage_options`` dictionary:
+
+.. code-block:: bash
+
+    storage_options = {
+        'key': "key.string",
+        'secret': "sercret.string",
+        'client_kwargs': {'endpoint_url': "https://uor-aces-o.s3-ext.jc.rl.ac.uk"},
+    }
+
+where ``key.string`` and ``secret.string`` are the key and secret strings needed to access the ``S3_BUCKET`` S3 bucket. Then,
+the call to ``Active`` is:
+
+.. code-block:: bash
+
+    active = Active("S3_BUCKET/ch330a.pc19790301-def.nc", ncvar='UM_m01s16i202_vn1106',
+                    storage_options=storage_options,
+                    active_storage_url="https://reductionist.jasmin.ac.uk/")
+
+Note that the `<https://reductionist.jasmin.ac.uk/>`_ specified here is the Reductionist server deployed
+on CEDA-JASMIN.
+
+Configuring a Reductionist deployment
+-------------------------------------
+
+A few points on how to deploy `Reductionist <https://github.com/stackhpc/reductionist-rs>`_ on a Rocky9 cloud machine:
+
+* 99% of the deployment documentation is found in `Reductionist's nice deployment instructions <https://stackhpc.github.io/reductionist-rs/deployment/>`_
+* there are a few caveats specific to a pristine Rocky9 (and other distros) deployment though:
+* (n00b step) always have a system ``pip`` by installing it with: ``python -m ensurepip --upgrade``
+* system Python executable is ``python3`` - you can, of course, ``ln -s`` it to ``python``, or, better, run Ansible pointing it to the correct system Python3:
+
+.. code-block:: bash
+
+    ansible-playbook -i reductionist-rs/deployment/inventory reductionist-rs/deployment/site.yml -e 'ansible_python_interpreter=/usr/bin/python3'
+
+* that call *may result* (as in our case) in an error:
+
+.. code-block:: bash
+
+    TASK [Ensure step RPM is installed] ****************************************************************************************************
+    fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for step-cli-0.24.4-1.x86_64: Package step-cli_0.24.4_amd643z16ickc.rpm is not signed"}
+
+that's because, in our case, we missed the ``step-cli`` package, and a ``dfn`` install is not well liked by the system (it's not ``mamba`` business);
+that gets sorted out via `Step's install docs <https://smallstep.com/docs/step-cli/installation>`_:
+
+.. code-block:: bash
+
+    wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.rpm
+    sudo rpm -i step-cli_amd64.rpm

doc/quickstart/configuration.rst

@@ -4,3 +4,48 @@
 Installation
 ************
 
+Conda-mamba environment
+-----------------------
+
+Use a Miniconda/Miniforge3 installer to create an environment using
+our conda ``environment.yml`` file; download the latest Miniconda3 for Linux installer from
+the `Miniconda project <https://docs.conda.io/en/latest/miniconda.html#linux-installers>`_,
+install it, then create and activate the PyActiveStorage environment:
+
+.. code-block:: bash
+
+    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+    bash Miniconda3-latest-Linux-x86_64.sh
+    (base) conda env create -n activestorage -f environment.yml
+    (base) conda activate activestorage
+
+.. note::
+
+    Our dependencies are all from ``conda-forge`` so there is no issue related
+    to the buggy (and paid-for) Anaconda main/defaults channel!
+
+Installing PyActiveStorage
+--------------------------
+
+The installation then can proceed: installing with ``pip`` and installing ``all`` (ie
+installing the development and test install):
+
+.. code-block:: bash
+
+    pip install -e .
+
+After installing, you can run tests via ``pytest -n 2``.
+
+Supported Python versions
+-------------------------
+
+We adhere to `SPEC0 <https://scientific-python.org/specs/spec-0000/>`_ and support the following Python versions:
+
+* 3.10
+* 3.11
+* 3.12
+* 3.13
+
+.. note::
+
+    PyActiveStorage is fully compatible with ``numpy >=2.0.0``.

doc/quickstart/installation.rst

@@ -4,3 +4,60 @@
 Running
 *******
 
+Example 1: S3 file
+------------------
+
+Here is a basic example of of obtaining a minumum on a slice ``[0:3, 4:6, 7:9]`` on the data for a variable
+``UM_m01s16i202_vn1106`` in a file ``ch330a.pc19790301-def.nc`` hosted on JASMIN's S3 storage in a bucket called ``bnl``:
+
+.. code-block:: python
+
+    import os
+
+    from activestorage.active import Active
+
+
+    def s3_file():
+        """Run a simple active storage instance test with S3 file."""
+        S3_BUCKET = "bnl"
+        storage_options = {
+            'key': "KEY",
+            'secret': "SECRET",
+            'client_kwargs': {'endpoint_url': "https://uor-aces-o.s3-ext.jc.rl.ac.uk"},
+        }
+
+        active_storage_url = "https://reductionist.jasmin.ac.uk/"
+        a_file = "ch330a.pc19790301-def.nc"
+
+        test_file_uri = os.path.join(
+            S3_BUCKET,
+            a_file
+        )
+
+        active = Active(test_file_uri, ncvar='UM_m01s16i202_vn1106',
+                        storage_options=storage_options,
+                        active_storage_url=active_storage_url)
+        active._version = 2
+        result = active.min[0:3, 4:6, 7:9]
+
+        return result  # 5098.625
+
+Example 2: HTTPS file
+---------------------
+
+Same as above, only the file is stored on an HTTPS-facing server (NGINX-enabled):
+
+.. code-block:: python
+
+    from activestorage.active import Active
+
+
+    def https_file():
+        """Run a simple active storage instance test with https file."""
+        test_file_uri = "https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.nc"
+
+        active = Active(test_file_uri, ncvar="cl")
+        active._version = 1
+        result = active.min[0:3, 4:6, 7:9]
+
+        return result  # numpy.array([0.6909787], dtype="float32")

doc/quickstart/running.rst

@@ -23,8 +23,7 @@ dependencies:
   - pytest-html !=2.1.0
   - pytest-metadata >=1.5.1
   - pytest-xdist
-  # Python packages needed for building docs
-  # re-add when we deploy the docs
-  # - autodocsumm >=0.2.2
-  # - sphinx >=5
-  # - sphinx_rtd_theme
+  # docs
+  - autodocsumm
+  - sphinx
+  - sphinx_rtd_theme

docs_Reductionist_deployment/deploy.md

@@ -50,27 +50,27 @@ requires-python = ">=3.10"
 
 [project.optional-dependencies]
 test = [
-    "pytest>6.0.0",
+    "pytest",
     "pytest-cov>=2.10.1",
     "pytest-html!=2.1.0",
+    "pytest-metadata>=1.5.1",
     "pytest-xdist",
     "dask",
     "moto",
 ]
-
-# very possible these will be needed
-# at some point in the near future
-# doc = [
-#     "sphinx>=6.1.3",
-#     "pydata_sphinx_theme",
-# ]
+# build docs
+doc = [
+    "autodocsumm",
+    "sphinx",
+    "sphinx_rtd_theme",
+]
+# to be added when functionality provided
 # develop = [
 #     "pre-commit",
 #     "pylint",
 #     "pydocstyle",
 # ]
 
-
 [project.urls]
 Code = "https://github.com/NCAS-CMS/PyActiveStorage"
 # Documentation = ""