Docs improvements (#411)

* Remove awkward "This class does xyz" immediately followed by
"Constructs an instance of..." stemming from the concatenation by Sphinx's
autoclass of class and constructor dosctrings. For a more detailed
explanation of the issues, see the comment in `.flake8` about D205 and D400.

* Added a target to make web docs locally rather than through sphinx-multiversion

* When building web docs, also build a dev version, which is treated as
latest in the version comparison. However, the version selected by default
is not dev, but the latest numeric one, unless dev is the only version
present.

* Render attributes more like properties.

* Some doc styling fixes:
  - smaller margin at the top of the definition in a definition list
  - bigger margin at the top of a chapter
  - code block caption larger and closer to code block
  - darker text color for method parameters

* Added queue_name and project_name to testing config and execparams as well
as the appropriate rendering in docs.

* Added a custom generator for the API doc, since sphinx autodoc lacks the
flexibility to produce nice results and manually doing the full api doc
is tedious and error prone.

This generator takes a file that represents a toc tree and outputs an api
doc that results in (roughly) the same toc tree, but with proper class,
module, and function docs. It also procudes a list of executors and
launchers, as well as their respective class docs, by querying the
appropriate psij executor/launcher list functions.

Last, it produces an index of all classes and functions.

* Clean output directory before building web docs, since leftovers may
interfere with the result.

* Needef for some CSS selectors.

* Show base class in class docs.

* Fix multiple targets for cross reference in method signature types.

* Pin spinx-tabs to 3.2.0

* This fixes a number of conceptual issues with executor registration:
   - allows for aliases (e.g., "pbs" -> "pbspro")
   - adds a "nice name" to executor descriptors (e.g., "rp" -> "Radical Pilot")
   - makes "pbs" the default rather than "pbspro"; "pbs" much predates the latter.
   - ensures that executor/launcher names are always stored and looked up
     in lower case

* Many docstring fixes

* Reformatting to be more in line with the rest of the code

* Unfortunately, sphinx gets confused when defining type aliases, so
revert to explicit types for the frankenpaths.

* Small ref fix

* Ref fix and use include for code example.

* Fix refs in add exec tutorial

* Updated arch and state diagram to be more in-line with the rest of the
styling as well as added a vector version that can be edited in, e.g.,
inkscape

* Updated user guide.

* Fixed spelling

* Turns out the adding and executor tutorial wasn't rendering at all.

* Another spelling fix

* Added docstrings for new props.

* Added a module loading example using pre_launch to the user guide

* Fixed odd spaces

* Fixed test function names

* Don't assume tests are run in the same directory as their source files.

* Fixed test

* Get output from subprocess

* This patch is not done on top of the new wait() semantics

* `join()` will ignore the first part if the last starts with a '/'

* It's `--utc` not `-utc`

* added readthedocs config file, which seems to be required now

* Fixed broken comments

* Spelling fixes

* Fixed readthedocs conf again

* Added requirements to rtd conf file

* Fixed missing import

* Removed unused import

* Updated `Job.wait` documentation to reflect the current semantics

Author: hategan

.flake8

@@ -36,11 +36,33 @@ max-line-length = 100
 # See https://www.python.org/dev/peps/pep-0008/#should-a-line-break-before-or-after-a-binary-operator
 # for a discussion about the issue and why we disable this.
 
-ignore = B902, D401, D100, W503
+# D205 - "1 blank line required between summary line and description"
+# D400 - "First line should end with a period"
+#
+# Sphinx (autoclass) has three modes of operation when it comes to
+# documenting classes: 'class', 'init', and 'both'. This pertains to how
+# docstrings for the class and init method are handled. Autoclass does
+# not produce a separate text for the class and init methods, but the
+# options above describe where the constructor documentation comes from.
+# This can be either the class docstring (autoclass_content = 'class')
+# or the __init__ docstring or both concatenated. The 'class' options
+# results in empty decriptions for the __init__ parameters. The
+# 'init' option ignores the class docstring, so, in order to get that
+# part into the output, it would need to be duplicated in the __init__
+# docstring. The better option is 'both'. However saying something like
+# "This class..." in the class docstring and "Initializes..." in the
+# __init__ docstring results in awkward phrasing when put together.
+# One can omit the description on the __init__ docstring and simply
+# document the parameters, but then flake8 complains about the
+# structure of the __init__ doctsring. The D205 and D400 errors are
+# precisely those complains. We disable these in order to have a
+# sane way of documenting classes with Sphinx' autoclass.
+
+
+ignore = B902, D205, D400, D401, D100, W503
 
 # D103 - Missing docstring in public function
 #
 # Ignore docstrings requirement in tests
 
 per-file-ignores = tests/*:D103
-

.readthedocs.yaml

@@ -0,0 +1,21 @@
+# Read the Docs configuration file for Sphinx projects
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+    
+# Set the OS, Python version and other tools you might need
+build:
+    os: ubuntu-22.04
+    tools:
+        python: "3.11"
+
+python:
+    install:
+        - requirements: requirements-docs.txt
+
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+    configuration: docs/conf.py
+    fail_on_warning: true

Makefile

@@ -57,7 +57,15 @@ docs:
 web-docs:
 	rm -rf docs/.generated
 	rm -rf docs/.web-build
+	git fetch --all --tags
 	PSIJ_WEB_DOCS=1 sphinx-multiversion docs docs/.web-build
+	PSIJ_WEB_DOCS=1 sphinx-build --keep-going -n -W -b html docs docs/.web-build/v/dev
+
+.PHONY: web-docs-dev
+web-docs-dev:
+	rm -rf docs/.generated
+	rm -rf docs/.web-build
+	PSIJ_WEB_DOCS=1 sphinx-build --keep-going -n -W -b html docs docs/.web-build/v/dev
 
 .PHONY: style
 style:

docs/_static/extras.js

@@ -35,6 +35,13 @@ $(document).ready(function() {
 
         initializeSelectors(selectorType, value);
     });
+    // Sphinx renders attributes differently from properties in that properties
+    // come with a nice "property" prefix in the heading, whereas attributes just
+    // have the name. This hacks the attribute headers to add a "attribute" text
+    // before the attribute name
+    $("dl.attribute > dt.sig").each(function() {
+        $(this).prepend("<em class=\"attribute\"><span class=\"pre\">attribute</span><span class=\"w\"></span></em>");
+    });
 });
 
 function detectAll(selectorType) {
@@ -70,12 +77,19 @@ function detectAll(selectorType) {
         if (text == '"<&' + selectorType + '>"') {
             $(this).addClass(selectorType + "-item").addClass("psij-selector-value");
         }
-        if (text == "executor" && prevSpans.length == 2
-            && prevSpans[0].text() == "execparams" && prevSpans[1].text() == '.') {
+        if (prevSpans.length == 2 && prevSpans[0].text() == "execparams" && prevSpans[1].text() == '.') {
             // remove <span>execparams</span> and <span>.</span>
             prevSpans[0].remove();
             prevSpans[1].remove();
-            $(this).addClass(selectorType + "-item").addClass("psij-selector-value");
+            if (text == "executor") {
+                $(this).addClass(selectorType + "-item").addClass("psij-selector-value");
+            }
+            else if (text == "queue_name") {
+                $(this).text("QUEUE_NAME");
+            }
+            else if (text == "project_name") {
+                $(this).text("PROJECT_NAME");
+            }
         }
         prevSpans.push($(this));
         if (prevSpans.length > 2) {

docs/_templates/layout.html

@@ -11,13 +11,13 @@
 {% endblock %}
 
 {% block header %}
-    <v-app id="psij-app">
+    <v-app id="psij-app" class="psij-docs">
         {% include "../../web/_includes/menu.html" %}
         {{ super() }}
 {% endblock %}
 
 {% block rootrellink %}
-    <li><a href="{{ pathto(theme_roottarget) }}">{{shorttitle|e}}</a>{% if versions %}{% include "../../web/_includes/versionselect.html" %}{% endif %}{{reldelim1}}</li>
+    <li><a href="{{ pathto(theme_roottarget) }}">{{shorttitle|e}}</a>{% include "../../web/_includes/versionselect.html" %}{{reldelim1}}</li>
 {% endblock %}
 
 

docs/api.rst

@@ -1,230 +1,6 @@
-The PSI/J API
+API Reference
 =============
 
-The most important classes in this library are ``Job`` and ``JobExecutor``,
-followed by ``Launcher``.
-
-The Job Class and Its Modifiers
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The Job-related classes listed in this section (``Job``, ``JobSpec``,
-``ResourceSpec``, and ``JobAttributes``) are independent of
-executor implementations. The authors strongly recommend that users
-program against these classes rather than adding executor-specific
-configuration options.
-
-.. autoclass:: psij.Job
-    :members:
-    :noindex:
-
-.. autoclass:: psij.JobStatus
-    :members:
-    :noindex:
-
-.. autoclass:: psij.JobState
-    :members:
-    :noindex:
-
-Job Modifiers
-^^^^^^^^^^^^^
-
-There can be a lot of configuration information that goes into each
-resource manager job, including its walltime, partition/queue, the number of nodes
-it needs, what kind of nodes, what quality of service the job requires, and
-so on.
-
-PSI/J splits those attributes into three groups: one for generic
-POSIX information, one for resource information, and one for resource manager
-scheduling policies.
-
-.. autoclass:: psij.JobSpec
-    :members:
-    :noindex:
-
-.. autoclass:: psij.ResourceSpec
-    :members:
-    :noindex:
-
-.. autoclass:: psij.JobAttributes
-    :members:
-    :noindex:
-
-
-.. _executors:
-
-Executors
-~~~~~~~~~
-
-Executors are concrete implementations of mechanisms that execute jobs.
-To get an instance of a specific executor, call
-:meth:`JobExecutor.get_instance(name) <psij.job_executor.JobExecutor.get_instance>`,
-with ``name`` being one of the installed executor names. Alternatively, directly
-instantiate the executor, e.g.:
-
-.. code-block:: python
-
-    from psij.executors.flux import FluxJobExecutor
-
-    ex = FluxJobExecutor()
-
-Rather than:
-
-.. code-block:: python
-
-    import psij
-
-    ex = psij.JobExecutor.get_instance('flux')
-
-Executors can be
-installed from multiple sources, so the precise list of executors
-available to a specific installation of the PSI/J Python library can vary.
-To get a list of available executors, run the following in a
-terminal:
-
-.. code-block:: shell
-
-    $ python -m psij plugins
-
-
-JobExecutor Base Class
-^^^^^^^^^^^^^^^^^^^^^^
-
-The ``psij.JobExecutor`` class is abstract, but offers concrete static methods
-for registering, fetching, and listing subclasses of itself.
-
-.. autoclass:: psij.job_executor.JobExecutor
-    :noindex:
-
-The concrete executor implementations provided by this version of PSI/J Python
-are:
-
-Cobalt
-""""""""""""""""""""""
-
-.. autoclass:: psij.executors.batch.cobalt.CobaltJobExecutor
-    :noindex:
-
-Flux
-""""""""""""""""""""""
-
-.. autoclass:: psij.executors.flux.FluxJobExecutor
-    :noindex:
-
-LSF
-""""""""""""""""""""""
-
-.. autoclass:: psij.executors.batch.lsf.LsfJobExecutor
-    :noindex:
-
-PBS
-""""""""""""""""""""""
-
-.. autoclass:: psij.executors.batch.pbspro.PBSProJobExecutor
-    :noindex:
-
-Slurm
-""""""""""""""""""""""
-
-.. autoclass:: psij.executors.batch.slurm.SlurmJobExecutor
-    :noindex:
-
-Local
-""""""""""""""""""""""
-
-.. autoclass:: psij.executors.local.LocalJobExecutor
-    :noindex:
-
-Radical Pilot
-""""""""""""""""""""""
-
-.. autoclass:: psij.executors.rp.RPJobExecutor
-    :noindex:
-
-.. _launchers:
-
-
-Launchers
-----------------
-
-Launchers are mechanisms to start the actual jobs on batch schedulers
-once a set of nodes has been allocated for the job. In essence, launchers
-are wrappers around the job executable which can provide additional
-features, such as setting up an MPI environment, starting a copy of the
-job executable on each allocated node, etc. 
-
-To get a launcher instance,
-call :meth:`Launcher.get_instance(name) <psij.launcher.Launcher.get_instance>`
-with ``name`` being the name of a launcher. Like job executors,
-launchers are plugins and can come from various places. To obtain a list
-of launchers, you can run:
-
-.. code-block:: shell
-
-    $ python -m psij plugins
-
-Launcher Base Class
-^^^^^^^^^^^^^^^^^^^
-
-Like the executor, the ``Launcher`` base class is abstract, but offers
-concrete static methods for registering and fetching subclasses of itself.
-
-.. autoclass:: psij.Launcher
-    :noindex:
-
-The PSI/J Python library comes with a core set of launchers, which are:
-
-aprun
-""""""""""""""""""""""
-
-.. autoclass:: psij.launchers.aprun.AprunLauncher
-    :members:
-    :noindex:
-
-jsrun
-""""""""""""""""""""""
-
-.. autoclass:: psij.launchers.jsrun.JsrunLauncher
-    :members:
-    :noindex:
-
-srun
-""""""""""""""""""""""
-
-.. autoclass:: psij.launchers.srun.SrunLauncher
-    :members:
-    :noindex:
-
-mpirun
-""""""""""""""""""""""
-
-.. autoclass:: psij.launchers.mpirun.MPILauncher
-    :members:
-    :noindex:
-
-single
-""""""""""""""""""""""
-
-.. autoclass:: psij.launchers.single.SingleLauncher
-    :members:
-    :noindex:
-
-multiple
-""""""""""""""""""""""
-
-.. autoclass:: psij.launchers.multiple.MultipleLauncher
-    :members:
-    :noindex:
-
-Other Package Contents
-----------------------
-
-.. automodule:: psij.exceptions
-    :members:
-    :noindex:
-
-
-API Reference
-----------------
 .. toctree::
-
-    .generated/modules
+    .generated/tree
+    .generated/index