add doc:links and docs:links:check

.github/workflows/checks.yml

@@ -39,6 +39,11 @@ jobs:
         run: |
           poetry run -- nox -s docs:build
 
+      - name: Link Check
+        run: |
+          poetry run -- nox -s docs:links:check
+
+
   Changelog:
     name: Changelog Update Check
     runs-on: ubuntu-24.04

doc/changes/unreleased.md

@@ -1,5 +1,35 @@
 # Unreleased
 
+## Summary
+
+### Links in the Documentation 
+This version of the PTB adds nox tasks to check links present in our documentation:
+
+    docs:link - List all the links within the documentation
+    docs:links:check - Checks whether all links in the documentation are accessible
+
+`docs:links:check` is run in the CI `checks.yml`. If this step fails in the CI,
+please check the output & manually resolve the issues. There might be some cases
+where you need to update your doc/conf.py with specific values for the allowed
+options for the [Linkcheck Builder](https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-the-linkcheck-builder).
+
+We recommend the following values be added:
+
+    linkcheck_rate_limit_timeout = 60
+    linkcheck_timeout = 15
+    linkcheck_delay = 30
+    linkcheck_retries = 2
+    linkcheck_anchors = False
+    linkcheck_ignore: list[str] = []
+    linkcheck_allowed_redirects = {
+        # All HTTP redirections from the source URI to
+        # the canonical URI will be treated as "working".
+        r"https://github\.com/.*": r"https://github\.com/login*"
+    }
+
+## ✨ Features
+* #409: Doc link & checks
+
 ## Refactoring
 * Switched deprecated Pydantic class-based `config` to `ConfigDict`
 

doc/conf.py

@@ -78,3 +78,15 @@
     "github_url": "https://github.com/exasol/python-toolbox",
     "accent_color": "grass",
 }
+# -- Configure link checking behavior  ----------------------------------------
+linkcheck_rate_limit_timeout = 60
+linkcheck_timeout = 15
+linkcheck_delay = 30
+linkcheck_retries = 2
+linkcheck_anchors = False
+linkcheck_ignore: list[str] = []
+linkcheck_allowed_redirects = {
+    # All HTTP redirections from the source URI to
+    # the canonical URI will be treated as "working".
+    r"https://github\.com/.*": r"https://github\.com/login*"
+}

doc/developer_guide/modules/sphinx/sphinx.rst

@@ -4,7 +4,7 @@ sphinx
 sphinx-multiversion
 +++++++++++++++++++
 
-The `sphinx-multiversion` extension is a modified copy of `Holzhaus/sphinx-multiversion <https://github.com/Holzhaus/sphinx-multiversion>`_. This copy was taken from version :code:`0.24.0`.
+The `sphinx-multiversion` extension is a modified copy of `sphinx-contrib/multiversion <https://github.com/sphinx-contrib/multiversion>`_. This copy was taken from version :code:`0.24.0`.
 
 It has been adjusted with minor code changes and modified defaults to work seamlessly with Exasol integration projects, which often require a specific project structure and layout. Additionally, it is designed to be used with an HTML theme that supports displaying and selecting multiple versions if the `versions` variable is set in the HTML context of sphinx. As of this writing, the theme used in conjunction with this modified version of `sphinx-multiversion` is `SHIBUYA <https://github.com/lepture/shibuya>`_, version :code:`2024.10.15`.
 

doc/github_actions/security_issues.rst

@@ -112,4 +112,4 @@ Ideas
 .. todo::
 
     Consider adapting common CVE report format as input, for additional details
-    `see here <https://github.com/CVEProject/cve-schema/blob/master/schema/v5.0/CVE_JSON_5.0_schema.json>`_.
+    `see here <https://github.com/CVEProject/cve-schema/blob/main/schema/CVE_Record_Format.json>`_.

doc/styleguide/guides/idioms/idioms.rst

@@ -37,5 +37,5 @@ where we picked it up from, rather than the "original" source.
 
 
 .. _Raymond Hettinger: https://github.com/rhettinger
-.. _Transform Code into Beautiful, Idiomatic Python: https://www.youtube.com/watch?v=OSGv2VnC0go>
+.. _Transform Code into Beautiful, Idiomatic Python: https://www.youtube.com/watch?v=OSGv2VnC0go
 .. _Transform Python Slides: https://speakerdeck.com/pyconslides/transforming-code-into-beautiful-idiomatic-python-by-raymond-hettinger-1

doc/styleguide/guides/style.rst

@@ -54,10 +54,10 @@ _____
 .. _Google Styleguide: https://google.github.io/styleguide/pyguide.html
 .. _PEP 8: https://peps.python.org/pep-0008/
 .. _Python Idioms: https://gist.github.com/0x4D31/f0b633548d8e0cfb66ee3bea6a0deff9
-.. _Python Like You Mean It: http://www.pythonlikeyoumeanit.com/module_2.html>
+.. _Python Like You Mean It: http://www.pythonlikeyoumeanit.com/module_2.html
 .. _Python Programming Idioms: https://en.wikibooks.org/wiki/Python_Programming/Idioms
 
-.. _Transform Code into Beautiful, Idiomatic Python: https://www.youtube.com/watch?v=OSGv2VnC0go>
+.. _Transform Code into Beautiful, Idiomatic Python: https://www.youtube.com/watch?v=OSGv2VnC0go
 .. _Transform Python Slides: https://speakerdeck.com/pyconslides/transforming-code-into-beautiful-idiomatic-python-by-raymond-hettinger-1
 .. _Stop Writing Classes: https://www.youtube.com/watch?v=o9pEzgHorH0
 .. _Refactoring Python: https://www.youtube.com/watch?v=D_6ybDcU5gc

doc/user_guide/features.rst

@@ -7,13 +7,13 @@ Uniform Project Layout
 PTB expects a default project layout following "convention over configuration" when possible and reasonable.
 See the cookie-cutter project template for details, which is part of the python-toolbox workspace and can be found in directory `project-template`.
 You can also generate a project from the template to explore the default structure.
-For more details on this, please check out section `"getting started" <getting_started.html>`_ section.
+For more details on this, please check out section :ref:`Getting Started` section.
 
 Nox
 ---
 
 The most central tool when interacting with the toolbox is :code:`nox`, which is the task runner used across all of Exasol's Python-based projects.
-The toolbox itself provides various standard tasks and a plugin mechanism to extend these tasks if needed. For more information regarding nox, please visit the `nox homepage <http://nox.thea.codes/en/stable/>`_.
+The toolbox itself provides various standard tasks and a plugin mechanism to extend these tasks if needed. For more information regarding nox, please visit the `nox homepage <https://nox.thea.codes/en/stable/>`_.
 
 Central files in regards to nox and the toolbox are:
 

doc/user_guide/getting_started.rst

@@ -1,3 +1,5 @@
+.. _Getting Started:
+
 Getting Started
 ===============
 

exasol/toolbox/nox/_documentation.py

@@ -1,9 +1,13 @@
 from __future__ import annotations
 
+import argparse
+import json
 import shutil
 import subprocess
 import sys
+import tempfile
 import webbrowser
+from pathlib import Path
 
 import nox
 from nox import Session
@@ -17,8 +21,6 @@
 
 def _build_docs(session: nox.Session, config: Config) -> None:
     session.run(
-        "poetry",
-        "run",
         "sphinx-build",
         "-W",
         "-b",
@@ -30,8 +32,6 @@ def _build_docs(session: nox.Session, config: Config) -> None:
 
 def _build_multiversion_docs(session: nox.Session, config: Config) -> None:
     session.run(
-        "poetry",
-        "run",
         "sphinx-multiversion",
         f"{config.doc}",
         DOCS_OUTPUT_DIR,
@@ -88,6 +88,92 @@ def clean_docs(_session: Session) -> None:
         shutil.rmtree(docs_folder)
 
 
+def _docs_list_links(doc_config: Path):
+    with tempfile.TemporaryDirectory() as path:
+        tmpdir = Path(path)
+        sp = subprocess.run(  # nosec
+            [
+                "sphinx-build",
+                "-b",
+                "linkcheck",
+                "-D",
+                "linkcheck_ignore=.*",
+                doc_config,
+                tmpdir,
+            ],
+            capture_output=True,
+            text=True,
+        )
+        if sp.returncode >= 2:
+            return sp.returncode, sp.stderr
+        output = tmpdir / "output.json"
+        links = output.read_text().split("\n")
+        file_links = []
+        for link in links:
+            if link != "":
+                line = json.loads(link)
+                if not line["uri"].startswith("#"):
+                    file_links.append(line)
+        file_links.sort(key=lambda file: file["filename"])
+        return 0, "\n".join(
+            f"filename: {fl['filename']}:{fl['lineno']} -> uri: {fl['uri']}"
+            for fl in file_links
+        )
+
+
+def _docs_links_check(doc_config: Path, args):
+    with tempfile.TemporaryDirectory() as path:
+        tmpdir = Path(path)
+        sp = subprocess.run(  # nosec
+            [
+                "sphinx-build",
+                "-b",
+                "linkcheck",
+                doc_config,
+                tmpdir,
+            ],
+        )
+        if args.output and sp.returncode <= 1:
+            result_json = tmpdir / "output.json"
+            dst = Path(args.output) / "link-check-output.json"
+            shutil.copyfile(result_json, dst)
+            print(f"file generated at path: {result_json.resolve()}")
+        return sp.returncode, (
+            None if sp.returncode >= 2 else (tmpdir / "output.txt").read_text()
+        )
+
+
+@nox.session(name="docs:links", python=False)
+def docs_list_links(session: Session) -> None:
+    """List all the links within the documentation."""
+    r_code, text = _docs_list_links(PROJECT_CONFIG.doc)
+    print(text)
+    if r_code != 0:
+        session.error()
+
+
+@nox.session(name="docs:links:check", python=False)
+def docs_links_check(session: Session) -> None:
+    """Checks whether all links in the documentation are accessible."""
+    parser = argparse.ArgumentParser(
+        prog="nox -s docs:links:check",
+        usage="nox -s docs:links:check -- [-h] [-o |--output]",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        "-o", "--output", type=Path, help="path to copy the output json", default=None
+    )
+    args = parser.parse_args(session.posargs)
+    r_code, problems = _docs_links_check(PROJECT_CONFIG.doc, args)
+    if r_code >= 2:
+        session.error(2)
+    if r_code == 1 or problems != "":
+        escape_red = "\033[31m"
+        print(escape_red + "errors:")
+        print(problems)
+        session.error(1)
+
+
 @nox.session(name="changelog:updated", python=False)
 def updated(_session: Session) -> None:
     """Checks if the change log has been updated"""

exasol/toolbox/templates/github/workflows/checks.yml

@@ -38,6 +38,10 @@ jobs:
         run: |
           poetry run -- nox -s docs:build
 
+      - name: Link Check
+        run: |
+          poetry run -- nox -s docs:links:check
+
   build-matrix:
     name: Generate Build Matrix
     uses: ./.github/workflows/matrix-python.yml

project-template/{{cookiecutter.repo_name}}/doc/conf.py

@@ -76,3 +76,16 @@
     "github_url": "https://github.com/exasol/{{cookiecutter.repo_name}}",
     "accent_color": "grass",
 }
+
+# -- Configure link checking behavior  ----------------------------------------
+linkcheck_rate_limit_timeout = 60
+linkcheck_timeout = 15
+linkcheck_delay = 30
+linkcheck_retries = 2
+linkcheck_anchors = False
+linkcheck_ignore: list[str] = []
+linkcheck_allowed_redirects = {
+    # All HTTP redirections from the source URI to
+    # the canonical URI will be treated as "working".
+    r"https://github\.com/.*": r"https://github\.com/login*"
+}

test/unit/documentation_test.py

@@ -0,0 +1,94 @@
+import shutil
+from unittest.mock import (
+    MagicMock,
+    patch,
+)
+
+import pytest
+
+import exasol.toolbox.nox._documentation
+from exasol.toolbox.nox._documentation import (
+    _docs_links_check,
+    _docs_list_links,
+)
+from noxconfig import Config
+
+
+@pytest.fixture()
+def file1():
+    return """
+https://examle.invalid
+:ref:`Test`"""
+
+
+@pytest.fixture()
+def index():
+    return """.. _Test:
+
+Test
+____
+
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   file"""
+
+
+@pytest.fixture()
+def expected1():
+    return """filename: file.rst:2 -> uri: https://examle.invalid"""
+
+
+def config(index, file, tmp_path):
+    test_doc = tmp_path / "doc"
+    test_doc.mkdir()
+    (test_doc / "_static").mkdir()
+    shutil.copyfile(Config.doc / "conf.py", test_doc / "conf.py")
+    rst_index = test_doc / "index.rst"
+    rst_file1 = test_doc / "file.rst"
+    rst_index.touch()
+    rst_file1.touch()
+    rst_index.write_text(index)
+    rst_file1.write_text(file)
+
+
+def test_docs_links(index, file1, expected1, tmp_path):
+    config(index, file1, tmp_path)
+    r_code, text = _docs_list_links(tmp_path / "doc")
+    assert (text == expected1) and not r_code
+
+
+@pytest.mark.parametrize(
+    "file2, expected2",
+    [
+        ("https://github.com/exasol/python-toolbox", (0, "")),
+        (
+            "http://nox.thea.codes/en/stable/",
+            (
+                0,
+                "file.rst:1: [redirected with Found] http://nox.thea.codes/en/stable/ to https://nox.thea.codes/en/stable/\n",
+            ),
+        ),
+        (
+            "https://github.com/exasol/python-toolbox/pull",
+            (
+                0,
+                "file.rst:1: [redirected permanently] https://github.com/exasol/python-toolbox/pull to https://github.com/exasol/python-toolbox/pulls\n",
+            ),
+        ),
+        (
+            "https://github.com/exasol/python-toolbox/asdf",
+            (
+                1,
+                "file.rst:1: [broken] https://github.com/exasol/python-toolbox/asdf: 404 Client Error: Not Found for url: https://github.com/exasol/python-toolbox/asdf\n",
+            ),
+        ),
+    ],
+)
+def test_docs_links_check(index, file2, expected2, tmp_path):
+    config(index, file2, tmp_path)
+    args = MagicMock
+    args.output = None
+    actual = _docs_links_check(tmp_path / "doc", args)
+    assert actual == expected2