Documentation/510 consolidate metrics and sonar

doc/changes/unreleased.md

@@ -1 +1,5 @@
 # Unreleased
+
+## Documentation
+
+* #510: Consolidated information of metrics & updated to include more about Sonar

doc/conf.py

@@ -88,5 +88,6 @@
 linkcheck_allowed_redirects = {
     # All HTTP redirections from the source URI to
     # the canonical URI will be treated as "working".
-    r"https://github\.com/.*": r"https://github\.com/login*"
+    r"https://github\.com/.*": r"https://github\.com/login*",
+    r"https://sonarcloud\.io.*": r"https://www.sonarsource\.com/products/sonarcloud/.*",
 }

doc/developer_guide/modules/modules.rst

@@ -5,5 +5,4 @@ Modules
     :maxdepth: 2
 
     sphinx/sphinx
-    nox
     nox_tasks

doc/index.rst

@@ -37,17 +37,11 @@ Documentation of the Exasol-Toolbox
 
         Document outlining the architectural and design principles and decisions in this project.
 
-    .. grid-item-card:: :octicon:`graph` Metrics
-        :link: metrics
-        :link-type: ref
-
-        Details on metrics collection and support related to and supported by the Python toolbox.
-
     .. grid-item-card:: :octicon:`question` FAQ
         :link: faq_toolbox
         :link-type: ref
 
-        Frequently asked questsions.
+        Frequently asked questions.
 
 
 .. toctree::
@@ -58,6 +52,5 @@ Documentation of the Exasol-Toolbox
    developer_guide/developer_guide
    tools
    github_actions/github_actions
-   metrics
    faq
    changes/changelog

doc/user_guide/features/index.rst

@@ -6,7 +6,7 @@ Features
 .. toctree::
     :maxdepth: 2
 
-    collecting_metrics
+    metrics/collecting_metrics
     creating_a_release
 
 Uniform Project Layout

doc/user_guide/features/metrics/collecting_metrics.rst

@@ -0,0 +1,90 @@
+Collecting metrics
+==================
+
+.. toctree::
+    :maxdepth: 2
+
+    project_report
+    sonar
+
+.. _direct_metrics:
+
+Direct metrics
+++++++++++++++
+
+The PTB allows you to collect various metrics on the quality of your project
+regarding Coverage, Security, and Static Code Analysis.
+
+For each metric, there is a dedicated nox session, generating one or multiple
+files and based on a selected external Python tool.
+
++------------------------------------+-----------------------------+--------------+
+| Nox session                        | Generated files             | Based on     |
++====================================+=============================+==============+
+| ``lint:code``                      | ``lint.txt``, ``lint.json`` | ``pylint``   |
++------------------------------------+-----------------------------+--------------+
+| ``lint:security``                  | ``.security.json``          | ``bandit``   |
++------------------------------------+-----------------------------+--------------+
+| ``test:unit -- --coverage``        | ``.coverage``               | ``coverage`` |
++------------------------------------+-----------------------------+--------------+
+| ``test:integration -- --coverage`` | ``.coverage``               | ``coverage`` |
++------------------------------------+-----------------------------+--------------+
+
+These metrics are computed for each point in your build matrix, e.g. for each
+Python version defined in the file ``noxconfig.py``:
+
+.. code-block:: python
+
+    @dataclass(frozen=True)
+    class Config:
+        python_versions = ["3.9", "3.10", "3.11", "3.12", "3.13"]
+
+The GitHub workflows of your project can:
+
+* Use a build matrix, e.g. using different Python versions as shown above
+* Define multiple test sessions, e.g. for distinguishing fast vs. slow or expensive tests.
+
+
+Reporting metrics
++++++++++++++++++
+
+Currently, within the PTB, we offer two methods by which the :ref:`direct_metrics`
+are summarized into reports:
+
+#. the nox session ``project:report``
+    This summarization tool can generate either a JSON or markdown summary where
+    it provides an overall coverage percentage, maintainability grade, and security
+    grade. The generated JSON follows a code-agnostic format which can be used
+    for code quality analysis across Exasol. The markdown summary can be displayed in
+    the GitHub Action's Summary for a given CI run. For more information, see :ref:`project_report`.
+
+#. SonarQube analysis
+    This summarization tool feeds into a feature-rich UI provided by
+    `Sonar <https://docs.sonarsource.com/sonarqube-server/latest/>`__. The total coverage
+    is calculated as a percentage and visually displayed per line on the altered code.
+    Security & linting issues are displayed in a list with links to the code & textual
+    descriptions on why they should be fixed and how they can be fixed. Additionally,
+    this information can be paired per project with a bot, which reports the code
+    quality analysis results within a PR. For further details, see :ref:`sonarqube_analysis`
+
+Both of these reporting options require that the generated files from the :ref:`direct_metrics`
+are existing and in the expected formats. As we have metrics for different Python
+versions, we pass on the metrics associated with the Python version named first in
+attribute ``python_versions`` of class ``Config`` to the reporting metrics tools.
+
+To perform this validation, we have defined two nox sessions. Due to the direct
+dependence on the nox session ``project:report`` and SonarQube Analysis on the
+aforementioned nox sessions, all of these are executed in succession in the CI's ``report.yml``.
+
++--------------------------+----------------------------------------------------------+
+| Nox session              | Actions                                                  |
++==========================+==========================================================+
+| ``artifacts:copy``       | * Combines coverage artifacts from various test sources  |
+|                          |   (unit, integration ...)                                |
+|                          | * Copies downloaded artifacts to their parent directory  |
++--------------------------+----------------------------------------------------------+
+| ``artifacts:validate``   | * Verifies that the ``.lint.json``, ``.lint.json``,      |
+|                          |   ``.security.json``, and ``.coverage`` are present      |
+|                          | * Checks that each file contains the expected attributes |
+|                          |   for that file type                                     |
++--------------------------+----------------------------------------------------------+

doc/user_guide/features/metrics/project_report.rst

@@ -0,0 +1,53 @@
+.. _project_report:
+
+``project:report``
+==================
+The nox session ``project:report`` provides an overall coverage percentage,
+maintainability grade, and security grade based on the :ref:`direct_metrics` collected.
+The definitions used for evaluating the quality of the Python code is defined in the
+`metrics.py`_ file, and the required fields are specified by the code-agnostic
+:ref:`metrics_schema` for Exasol. This nox session can return its analysis in two forms:
+
+* JSON
+    This directly meets the requirements for the :ref:`metrics_schema`. In our CI runs,
+    a JSON is created & uploaded as an artifact, which can be downloaded later by the
+    crawler project.
+* markdown
+    This is displayed in the GitHub Action's Summary for a given CI run. Displaying
+    this content per CI run gives the developer immediate feedback as to how the code
+    quality has changed between code modifications.
+
+
+.. _metrics_schema:
+
+Metrics schema
+++++++++++++++
+The metrics schema has been established to provide a unified approach for metrics and
+reporting across our projects. More details on its motivation, development, & usage
+can be found in the following resources:
+
+* `Exasol Schemas`_
+* `Metrics Schema`_
+* `Metrics Schema Project`_
+
+For our open-source projects, there is a scheduled job that regularly collects metrics
+from projects. This data is then aggregated and added to a central data store. For more
+details, please refer to the crawler project documentation.
+
+Development
+-----------
+
+If the metrics schema needs to be updated, the `Metrics Schema Project`_ provides a
+convenient way (via a Pydantic model) to update and generate an updated schema for the
+metrics.
+
+.. note::
+
+   The updated version needs to be first integrated into the `Exasol Schemas Project`_.
+
+
+.. _Exasol Schemas: https://schemas.exasol.com
+.. _Exasol Schemas Project: https://github.com/exasol/schemas
+.. _Metrics Schema: https://schemas.exasol.com/project-metrics-0.2.0.html
+.. _metrics.py: https://github.com/exasol/python-toolbox/blob/main/exasol/toolbox/metrics.py
+.. _Metrics Schema Project: https://github.com/exasol/python-toolbox/tree/main/metrics-schema