Documentation/510 consolidate metrics and sonar

doc/changes/unreleased.md

@@ -3,3 +3,4 @@
 ## Documentation
 
 * #512: Consolidated and added to deploying documentation pages
+* #510: Consolidated information of metrics & updated to include more about Sonar

doc/conf.py

@@ -90,4 +90,5 @@
     # the canonical URI will be treated as "working".
     r"https://github\.com/.*": r"https://github\.com/login*",
     r"https://www.sphinx-doc\.org/.*": r"https://www.sphinx-doc\.org/en/master/.*",
+    r"https://sonarcloud\.io.*": r"https://www.sonarsource\.com/products/sonarcloud/.*",
 }

doc/developer_guide/modules/modules.rst

@@ -4,5 +4,4 @@ Modules
 .. toctree::
     :maxdepth: 2
 
-    nox
     nox_tasks

doc/index.rst

@@ -37,17 +37,11 @@ Documentation of the Exasol-Toolbox
 
         Document outlining the architectural and design principles and decisions in this project.
 
-    .. grid-item-card:: :octicon:`graph` Metrics
-        :link: metrics
-        :link-type: ref
-
-        Details on metrics collection and support related to and supported by the Python toolbox.
-
     .. grid-item-card:: :octicon:`question` FAQ
         :link: faq_toolbox
         :link-type: ref
 
-        Frequently asked questsions.
+        Frequently asked questions.
 
 
 .. toctree::
@@ -58,6 +52,5 @@ Documentation of the Exasol-Toolbox
    developer_guide/developer_guide
    tools
    github_actions/github_actions
-   metrics
    faq
    changes/changelog

doc/user_guide/features/index.rst

@@ -6,7 +6,7 @@ Features
 .. toctree::
     :maxdepth: 2
 
-    collecting_metrics
+    metrics/collecting_metrics
     creating_a_release
     documentation/index
 

doc/user_guide/features/metrics/collecting_metrics.rst

@@ -0,0 +1,82 @@
+Collecting metrics
+==================
+
+.. toctree::
+    :maxdepth: 2
+
+    project_report
+    sonar
+
+.. _generated_metrics:
+
+Generated metrics
++++++++++++++++++
+
+The PTB allows you to collect various metrics on the quality of your project
+regarding Coverage, Security, and Static Code Analysis.
+
+For each metric, there is a dedicated nox session, generating one or multiple
+files and based on a selected external Python tool.
+
++------------------------------------+-----------------------------+--------------+
+| Nox session                        | Generated files             | Based on     |
++====================================+=============================+==============+
+| ``lint:code``                      | ``lint.txt``, ``lint.json`` | ``pylint``   |
++------------------------------------+-----------------------------+--------------+
+| ``lint:security``                  | ``.security.json``          | ``bandit``   |
++------------------------------------+-----------------------------+--------------+
+| ``test:unit -- --coverage``        | ``.coverage``               | ``coverage`` |
++------------------------------------+-----------------------------+--------------+
+| ``test:integration -- --coverage`` | ``.coverage``               | ``coverage`` |
++------------------------------------+-----------------------------+--------------+
+
+These metrics are computed for each element in your build matrix, e.g. for each
+Python version defined in the file ``noxconfig.py``:
+
+.. code-block:: python
+
+    @dataclass(frozen=True)
+    class Config:
+        python_versions = ["3.9", "3.10", "3.11", "3.12", "3.13"]
+
+The GitHub workflows of your project can:
+
+* Use a build matrix, e.g. using different Python versions as shown above
+* Define multiple test sessions, e.g. for distinguishing fast vs. slow or expensive tests.
+
+
+Reporting metrics
++++++++++++++++++
+
+Currently, the PTB offers two methods to aggregate the :ref:`generated_metrics`
+into a report:
+
+#. the nox session ``project:report``
+    This is an Exasol-specific summarization tool. For more information, see :ref:`project_report`.
+
+#. SonarQube analysis
+    This summarization tool feeds into a feature-rich UI provided by
+    `Sonar <https://docs.sonarsource.com/sonarqube-server/latest/>`__. For further
+    details, see :ref:`sonarqube_analysis`
+
+Both of these reporting options require that the generated files from the :ref:`generated_metrics`
+are existing and in the expected formats. As there are metrics for different Python
+versions, the PTB uses only the metrics associated with the Python version named first
+in the attribute ``python_versions`` of class ``Config`` to the reporting metrics tools.
+
+To perform this validation, there are two nox sessions. Due to the direct
+dependence on the nox session ``project:report`` and SonarQube Analysis on the
+aforementioned nox sessions, all of these are executed in succession in the CI's ``report.yml``.
+
++--------------------------+----------------------------------------------------------+
+| Nox session              | Actions                                                  |
++==========================+==========================================================+
+| ``artifacts:copy``       | * Combines coverage artifacts from various test sources  |
+|                          |   (unit, integration ...)                                |
+|                          | * Copies downloaded artifacts to their parent directory  |
++--------------------------+----------------------------------------------------------+
+| ``artifacts:validate``   | * Verifies that the ``.lint.json``, ``.lint.txt``,       |
+|                          |   ``.security.json``, and ``.coverage`` are present      |
+|                          | * Checks that each file contains the expected attributes |
+|                          |   for that file type                                     |
++--------------------------+----------------------------------------------------------+

doc/user_guide/features/metrics/project_report.rst

@@ -0,0 +1,48 @@
+.. _project_report:
+
+``project:report``
+==================
+After collecting the metrics described in by :ref:`generated_metrics`, you can use the
+nox session ``project:report`` to create a report using one of the following formats:
+
+* JSON
+    This format is usually used in the CI. The created JSON file is uploaded as an
+    artifact, which can be downloaded and aggregated for multiple projects
+    (see :ref:`metrics_schema`).
+* markdown
+    This is displayed in the GitHub Action's summary for a given CI run. Displaying
+    this content per CI run gives the developer immediate feedback as to how the code
+    quality has changed between code modifications.
+
+
+.. _metrics_schema:
+
+Metrics schema
+++++++++++++++
+The PTB supports the uniform json schema for metrics used by all projects
+of the Exasol Product Integration Team:
+
+* `Exasol Schemas`_
+* `Metrics Schema`_
+* `Metrics Schema Project`_
+
+The Integration team runs regular aggregation of the metrics from all projects into a centralized data sink
+as decribed in the company Wiki.
+
+Development
+-----------
+
+If the metrics schema needs to be updated, the `Metrics Schema Project`_ provides a
+convenient way (via a Pydantic model) to update and generate an updated schema for the
+metrics.
+
+.. note::
+
+   The updated version needs to be first integrated into the `Exasol Schemas Project`_.
+
+
+.. _Exasol Schemas: https://schemas.exasol.com
+.. _Exasol Schemas Project: https://github.com/exasol/schemas
+.. _Metrics Schema: https://schemas.exasol.com/project-metrics-0.2.0.html
+.. _metrics.py: https://github.com/exasol/python-toolbox/blob/main/exasol/toolbox/metrics.py
+.. _Metrics Schema Project: https://github.com/exasol/python-toolbox/tree/main/metrics-schema