Improve dev documentation (#835)

Author: sandorkertesz

docs/development.rst

@@ -0,0 +1,24 @@
+.. _conda_setup:
+
+Development setup with Conda
+------------------------------
+
+You can set up a development environment using Conda. Below are the instructions to do so.
+
+First, clone the repository locally. You can use the following command:
+
+.. code-block:: shell
+
+   git clone --branch develop git@github.com:ecmwf/earthkit-data.git
+
+
+Next, enter your git repository and run the following commands:
+
+.. code-block:: shell
+
+    make conda-env-update
+    conda activate earthkit-data
+    make setup
+    pip install -e .
+
+This will create a new conda environment called "earthkit-data" with all the dependencies installed into it. This setup enables the `pre-commit`_ hooks, performing a series of quality control checks on every commit. If any of these checks fails the commit will be rejected.

docs/development/conda.rst

@@ -0,0 +1,34 @@
+.. _dev_docs:
+
+
+Documentation
+-------------------
+
+Building the documentation locally
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To build the documentation locally, please install the Python dependencies first:
+
+.. code-block:: shell
+
+    cd docs
+    pip install -r requirements.txt
+
+Then the documentation can be built by running the following command from the ``docs`` folder:
+
+.. code-block:: shell
+
+    make html
+
+To see the generated HTML documentation open the ``docs/_build/html/index.html`` file in your browser.
+
+Notebook examples
+~~~~~~~~~~~~~~~~~~~~~~
+
+The notebook examples are located in the ``docs/examples`` folder and are listed in ``docs/examples/index.rst``.
+
+
+If you add a new notebook example please:
+
+- also add it to ``docs/examples/index.rst`` so that it appears in the documentation.
+- ensure the title and the subchapter headings have the same size as in the other notebook. Title with ``##``, subchapter with ``###``.

docs/development/docs.rst

@@ -0,0 +1,21 @@
+Development
+===========
+
+
+The code repository is hosted on `Github`_, testing, bug reports and contributions are highly welcomed and appreciated. Feel free to fork it and submit your PRs against the **develop** branch.
+
+
+Development guide
+~~~~~~~~~~~~~~~~~
+
+.. toctree::
+   :maxdepth: 1
+
+   setup
+   conda
+   tests
+   docs
+
+
+
+.. _`Github`: https://github.com/ecmwf/earthkit-data

docs/development/index.rst

@@ -0,0 +1,44 @@
+.. _dev_setup:
+
+Development setup with virtualenv
+-----------------------------------
+
+The recommended development environment is a **Python virtual environment**.
+
+First, clone the repository locally. You can use the following command:
+
+.. code-block:: shell
+
+   git clone --branch develop git@github.com:ecmwf/earthkit-data.git
+
+
+Next, create your Python virtual environment and activate it. E.g. assuming your virtual environment is called `earthkit-dev` you can do:
+
+.. code-block:: shell
+
+   cd YOUR_VENVS_DIR
+   python -m venv earthkit-dev
+   source earthkit-dev/bin/activate
+
+
+Next, install your repo with the development dependencies in editable mode by running the following commands from the root of the repository:
+
+.. code-block:: shell
+
+   pip install -e .[dev]
+
+
+When using zsh you might need to quote the square brackets like this:
+
+.. code-block:: shell
+
+    pip install -e ."[dev]"
+
+We strongly recommend using the `pre-commit`_ hooks for the developments. These hooks perform a series of quality control checks on every commit. If any of these checks fails the commit will be rejected. To install the hooks run the following commands in the root of the repository:
+
+.. code-block:: shell
+
+    pip install pre-commit
+    pre-commit install
+
+.. _`pre-commit`: https://pre-commit.com/

docs/development/setup.rst

@@ -0,0 +1,83 @@
+.. _testing:
+
+Testing
+-----------------------
+
+To run the test suite, you can use the following command:
+
+.. code-block:: shell
+
+    pytest
+
+
+Long tests
+~~~~~~~~~~~~~
+
+Please note that by default all the tests based on remote services e.g. :ref:`data-sources-mars` are skipped. This is done because they can take a very long time to complete or just hang. To enable all these tests you need to run:
+
+.. code-block:: shell
+
+    pytest -E long -v
+
+If just want to run e.g. the :ref:`data-sources-mars` tests you can use:
+
+.. code-block:: shell
+
+    pytest -E long -v -k mars
+
+
+Timeout for CDS tests
+~~~~~~~~~~~~~~~~~~~~~~
+
+Some :ref:`data-sources-cds` retrieval tests used to hang so an execution timeout was added to all the tests in ``tests/sources/test_cds.py``. The default value is 30 seconds and it can be controlled via the ``--cds-timeout`` custom option to ``pytest``. Please note that some tests have a custom hardcoded timeout value that cannot be changed via this option.
+
+E.g. to set the timeout to 60 seconds for all CDS tests run (supposing their names contain "test_cds"):
+
+.. code-block:: shell
+
+    pytest -E long --cds-timeout=60 -v -k test_cds
+
+
+Credentials
+~~~~~~~~~~~~~~~
+
+Some tests require credentials to access remote services. The existence of credentials are checked and the related tests are skipped accordingly. The logic can be found in ``src/earthkit/data/testing.py``.
+
+FDB tests
+~~~~~~~~~~~~
+
+The :ref:`data-sources-fdb` tests are only run if ``pyfdb`` can be imported and the ``FDB_HOME`` environment variable is set. See the logic in ``src/earthkit/data/testing.py``.
+
+
+Optional dependencies
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Tests generally require all optional dependencies to be installed. However, the availability of some optional dependencies are checked and the related tests are skipped accordingly. This is typically used for dependencies which are not part of the ``[all]`` install option (see :ref:`install`). The logic can be found in ``src/earthkit/data/testing.py``.
+
+
+no_cache_init tests
+~~~~~~~~~~~~~~~~~~~
+
+Tests marked with ``pytest.mark.no_cache_init`` use the ``pytest-forked`` plugin to run in a separate process. They must be run as:
+
+.. code-block:: shell
+
+    pytest --forked -v -m no_cache_init
+
+
+Notebooks
+~~~~~~~~~~~~~
+
+The notebook examples from the ``docs/examples`` folder are part of the test suite and are run automatically. However, some notebooks are skipped mainly because they use remote data sources. This is controlled via the ``SKIP`` list in ``tests/documentation/test_notebooks.py``. You can modify this list to add or remove notebooks to be skipped.
+
+To run only the notebooks tests you can use:
+
+.. code-block:: shell
+
+    pytest -v -m notebook
+
+
+Documentation code snippets
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All ``.py`` files in the ``docs`` folder can potentially contain example code snippets and are part of the test suite and run automatically. Many of these are actually not examples and skipped. This is controlled via the ``SKIP`` list in ``tests/documentation/test_examples.py``. You can modify this list to add or remove snippets to be skipped.

docs/development/tests.rst

@@ -56,7 +56,7 @@ Quick start
    howtos
    guide/index
    api
-   development
+   development/index
 
 .. toctree::
    :maxdepth: 1

docs/index.rst

@@ -15,7 +15,7 @@
 
 from earthkit.data.testing import earthkit_file
 
-IGNORE = [
+SKIP = [
     "conf.py",
     "xml2rst.py",
     "actions.py",
@@ -37,7 +37,7 @@ def example_list():
     for root, _, files in os.walk(EXAMPLES):
         for file in files:
             path = os.path.join(root, file)
-            if path.endswith(".py") and file not in IGNORE:
+            if path.endswith(".py") and file not in SKIP:
                 n = len(EXAMPLES) + 1
                 examples.append(path[n:])