Replace pydoc generation with mkdoc (#410)

Changes:

* Completely replaces pydoc with mkdoc. This is done to allow more expressiveness in docstrings. They now can include code-blocks, links to other functions, all the sort of snazzy things that come from rendering docstrings natively with rst.
* Moves the py client docs from a directory in the main documentation to its own page (which links back to the original page). Same idea as the rest api documentation.
* Removes the link to py client documentation in the user section. Now the only way to get to it is from the top menu.

Author: voetberg

docs/index.md

@@ -61,8 +61,8 @@ Developer documentation will help you get started. Peruse some common REST API &
 Client API references that are directly derived from Rucio's python
 libraries. We also have a contribution guide for those who wish to pitch in.
 
-- [Client API Documentation](client_api/accountclient)
-- REST API Documentation
+- [Client API Documentation](pathname:///html/site/client.html)
+- [REST API Documentation](pathname:///html/rest_api_doc.html)
 - [Contributing guide](contributing)
 
 ## Contributing to the Documentation

docs/user/developing_with_rucio.md

@@ -18,7 +18,7 @@ rucio_client.ping()
 ```
 
 The methods are separated per resource type. The API in full can be viewed
-[here](/client_api/accountclient).
+[here](pathname:///html/site/client.html).
 
 ### Errors and Exceptions
 

tools/build_documentation.sh

@@ -35,7 +35,7 @@ mkdir -p "$SCRIPT_DIR"/../website/static/html/
 
 cp -r "$AUTO_GENERATED"/rest_api_doc_spec.yaml "$SCRIPT_DIR"/../website/static/yaml/
 cp -r "$AUTO_GENERATED"/rest_api_doc.html "$SCRIPT_DIR"/../website/static/html/
-cp -r "$AUTO_GENERATED"/client_api "$DOCS"
+cp -r "$AUTO_GENERATED"/site "$SCRIPT_DIR"/../website/static/html/
 cp -r "$AUTO_GENERATED"/bin "$DOCS"
 
 

tools/run_in_docker/generate_client_api_docs.sh

@@ -4,22 +4,11 @@ set -e
 
 echo "Generating the Client Api Docs..."
 
-pip install --upgrade "pydoc-markdown>3" &> /dev/null
+pip install --upgrade mkdocs mkdocs-gen-files mkdocstrings-python mkdocs-material
 
 mkdir -p /auto_generated/client_api
-for f in rucio/lib/rucio/client/*.py; do
-    if [[ $f =~ "__init__" ]] || [[ $f =~ "richclient" ]]; then
-	continue
-    fi
 
-    executable_name=$(basename "$f" ".py")
+python3 generate_client_api_pages.py
+mkdocs build --clean --no-directory-urls
 
-    config="
-processors:
-  - type: rucio_processor.RucioProcessor
-renderer:
-  type: rucio_renderer.RucioRenderer"
-    content=$(PYTHONPATH=. pydoc-markdown -I rucio/lib/rucio/client -m "$executable_name" "$config")
-
-    echo "$content" > /auto_generated/client_api/"$executable_name".md
-done
+cp -r site /auto_generated/

tools/run_in_docker/generate_client_api_pages.py

@@ -0,0 +1,31 @@
+"""Generate the code reference pages and navigation."""
+
+import os
+from pathlib import Path
+
+import mkdocs_gen_files
+
+src = Path("rucio/lib/rucio/client/")
+root = src.parent
+doc_path = "/auto_generated/client_api"
+files = os.listdir(doc_path)
+for file in files:
+    os.remove(f"{doc_path}/{file}")
+
+
+py_files = [f for f in list(src.rglob("*.py")) if "__init__.py" not in f.name]
+
+for path in py_files:
+    module_name = path.name.split(".py")[0]
+
+    try:
+        full_doc_path = Path(f"{doc_path}/{module_name}.md")
+        __import__(f"rucio.client.{module_name}")
+        with mkdocs_gen_files.open(full_doc_path, "a") as fd:
+            module_path = path.relative_to(src).with_suffix("")
+            fd.write(f"::: rucio.client.{module_path}")
+            fd.write("\n\n")
+
+    except ImportError:
+        print(f"Could not access client page for {module_name}")
+        pass

tools/run_in_docker/mkdocs.yml

@@ -0,0 +1,48 @@
+site_name: "Rucio Python Client Documentation"
+repo_url: https://github.com/rucio/rucio/
+site_url: https://rucio.cern.ch/documentation/
+copyright: "Copyright © 2024 CERN"
+docs_dir: /auto_generated/client_api
+use_directory_urls: true
+not_in_nav: 
+  /auto_generated
+
+theme: 
+  name: material
+  logo: ../../img/rucio_horizontaled_black_cropped.svg
+  extra_css: ../../../src/customTheme.css
+  extra: 
+    homepage: /documentation/
+  features:
+  - content.code.copy
+  - toc.follow
+  palette: 
+    - scheme: default
+      primary: white
+      toggle:
+        icon: material/brightness-7 
+        name: Switch to dark mode
+    - scheme: slate
+      primary: white
+      toggle:
+        icon: material/brightness-4
+        name: Switch to light mode
+
+markdown_extensions: 
+- toc:
+    permalink: true
+plugins:
+- search
+- autorefs
+- mkdocstrings:
+    default_handler: python
+    handlers:
+      python:
+        options:
+          members_order: "source"
+          heading_level: 2
+          show_source: false
+          allow_inspection: false
+          merge_init_into_class: true
+          docstring_style: "sphinx"
+          show_bases: false