diff --git a/doc/_static/metrics-artifact.png b/doc/_static/metrics-artifact.png deleted file mode 100644 index e9dd190d0..000000000 Binary files a/doc/_static/metrics-artifact.png and /dev/null differ diff --git a/doc/_static/metrics-workflow-summary.png b/doc/_static/metrics-workflow-summary.png deleted file mode 100644 index 5efe68a64..000000000 Binary files a/doc/_static/metrics-workflow-summary.png and /dev/null differ diff --git a/doc/_static/nothing-to-see-here.png b/doc/_static/nothing-to-see-here.png deleted file mode 100644 index 23da50fa3..000000000 Binary files a/doc/_static/nothing-to-see-here.png and /dev/null differ diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md index 08da90278..5e46a24fd 100644 --- a/doc/changes/unreleased.md +++ b/doc/changes/unreleased.md @@ -3,3 +3,4 @@ ## Documentation * #512: Consolidated and added to deploying documentation pages +* #510: Consolidated information of metrics & updated to include more about Sonar diff --git a/doc/conf.py b/doc/conf.py index 6244d64f3..a872058dc 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -90,4 +90,5 @@ # the canonical URI will be treated as "working". r"https://github\.com/.*": r"https://github\.com/login*", r"https://www.sphinx-doc\.org/.*": r"https://www.sphinx-doc\.org/en/master/.*", + r"https://sonarcloud\.io.*": r"https://www.sonarsource\.com/products/sonarcloud/.*", } diff --git a/doc/developer_guide/modules/modules.rst b/doc/developer_guide/modules/modules.rst index c38340dec..21c47bf28 100644 --- a/doc/developer_guide/modules/modules.rst +++ b/doc/developer_guide/modules/modules.rst @@ -4,5 +4,4 @@ Modules .. toctree:: :maxdepth: 2 - nox nox_tasks diff --git a/doc/developer_guide/modules/nox.rst b/doc/developer_guide/modules/nox.rst deleted file mode 100644 index 20966422c..000000000 --- a/doc/developer_guide/modules/nox.rst +++ /dev/null @@ -1,10 +0,0 @@ -nox -=== - -The nox package contains nox related functionalities like pre defined nox tasks. - -.. figure:: ../../_static/nothing-to-see-here.png - :alt: nothing to see here - :target: https://imgflip.com/i/2a7gqa - - source: `imgflip.com `_ diff --git a/doc/index.rst b/doc/index.rst index ac7b7d291..cc6098a2e 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -37,17 +37,11 @@ Documentation of the Exasol-Toolbox Document outlining the architectural and design principles and decisions in this project. - .. grid-item-card:: :octicon:`graph` Metrics - :link: metrics - :link-type: ref - - Details on metrics collection and support related to and supported by the Python toolbox. - .. grid-item-card:: :octicon:`question` FAQ :link: faq_toolbox :link-type: ref - Frequently asked questsions. + Frequently asked questions. .. toctree:: @@ -58,6 +52,5 @@ Documentation of the Exasol-Toolbox developer_guide/developer_guide tools github_actions/github_actions - metrics faq changes/changelog diff --git a/doc/metrics.rst b/doc/metrics.rst deleted file mode 100644 index 9fdb315a4..000000000 --- a/doc/metrics.rst +++ /dev/null @@ -1,123 +0,0 @@ -.. _metrics: - -:octicon:`graph` Metrics -======================== - -To establish a unified approach for metrics and reporting across our projects, we have developed a simple and straightforward metrics schema, see: `Exasol Schemas`_ (Section: *Exasol Project Metrics*). -This schema is designed for ease of use and can be adopted universally across our projects for reporting fundamental metric data. - -Why a Custom Format? --------------------- - -#. **Simplicity and Clarity**: - - - The schema is straightforward, making it easy to understand and implement. - -#. **Vendor Neutrality and Extensibility**: - - - It is not tied to any specific vendor, allowing us to customize and extend it as needed. - -#. **Ease of Integration**: - - - Existing tools and vendor solutions can be seamlessly integrated by creating mappings from vendor results to our metrics schema. - -#. **Customizable Metrics Generation and Evaluation**: - - - We have the flexibility to adjust how metric data is generated and evaluated based on our specific needs. This, for example, allows us the freedom to rate maintainability differently across various language ecosystems, avoiding lock-in to a single method. - - -Schema ------- - -see also `Exasol Schemas`_. - -Examples -________ - -**0.1.0** - -.. code-block:: json - - { - "commit": "99654dfcbd6169f6c78e4f5117a3184891d36f5d", - "date": "2024-10-25 10:43:10.145235", - "coverage": 33.910533910533914, - "maintainability": "B", - "reliability": "N/A", - "security": "F", - "technical_debt": "N/A" - } - -**0.2.0** - -.. code-block:: json - - { - "project": "python-toolbox", - "commit": "99654dfcbd6169f6c78e4f5117a3184891d36f5d", - "date": "2024-10-25 10:43:10.145235", - "coverage": 33.910533910533914, - "maintainability": "B", - "reliability": "N/A", - "security": "F", - "technical_debt": "N/A" - } - - - - - -Workflows ---------- - -The Python toolbox comes with various tools to help maintain our Python projects. One of these tools are workflow templates, which can be generated and updated by default. The toolbox also provides a workflow called *report*, which gathers and reports a metrics summary for the project's CI run. For details on the workflow, run :code:`tbx workflow show report` to see the full workflow definitions. Below, you can see two screenshots of the results and artifacts this workflow provides. - -.. figure:: _static/metrics-workflow-summary.png - :alt: Metrics Workflow Summary - -.. figure:: _static/metrics-artifact.png - :alt: Metrics Artifact - -Tooling -------- - -Projects -________ - -Projects can run the nox task :code:`nox -s project:report` to generate their project metrics if they are using the :code:`exasol-toolbox`. - - -Development -___________ - -If the metrics schema needs to be updated, the `Metrics Schema Project`_ provides a convenient way (Pydantic model-based) to update and generate an updated schema for the metrics. - -.. note:: - - The updated version needs to be integrated into the `Exasol Schemas Project`_. - - -Ratings -------- -The strategy on how the rating of the Python code and projects work can be seen in the `metrics.py`_ file. - - -Collection ----------- -For our open-source projects, there is a scheduled job that regularly collects metrics information from projects. This data is then aggregated and added to a central data store. For more details, please refer to the crawler project documentation. - - -Links & References ------------------- -* `Exasol Schemas`_ -* `Metrics Schema`_ -* `Metrics Schema Project`_ -* `metrics.py`_ - - -.. _Exasol Schemas: https://schemas.exasol.com -.. _Exasol Schemas Project: https://github.com/exasol/schemas -.. _Metrics Schema: https://schemas.exasol.com/project-metrics-0.2.0.html -.. _metrics.py: https://github.com/exasol/python-toolbox/blob/main/exasol/toolbox/metrics.py -.. _Metrics Schema Project: https://github.com/exasol/python-toolbox/tree/main/metrics-schema - diff --git a/doc/user_guide/features/collecting_metrics.rst b/doc/user_guide/features/collecting_metrics.rst deleted file mode 100644 index 7fa9ef70b..000000000 --- a/doc/user_guide/features/collecting_metrics.rst +++ /dev/null @@ -1,37 +0,0 @@ -Collecting metrics -================== - -The PTB allows you to collect various metrics on the quality of your project -regarding Coverage, Security, and Static Code Analysis. - -For each metric, there is a dedicated nox session, generating one or multiple -files and based on a selected external Python tool. - -+------------------------------------+-----------------------------+--------------+ -| Nox session | Generated files | Based on | -+====================================+=============================+==============+ -| ``lint:code`` | ``lint.txt``, ``lint.json`` | ``pylint`` | -+------------------------------------+-----------------------------+--------------+ -| ``lint:security`` | ``.security.json`` | ``bandit`` | -+------------------------------------+-----------------------------+--------------+ -| ``test:unit -- --coverage`` | ``.coverage`` | ``coverage`` | -+------------------------------------+-----------------------------+--------------+ -| ``test:integration -- --coverage`` | ``.coverage`` | ``coverage`` | -+------------------------------------+-----------------------------+--------------+ - -These metrics are computed for each point in your build matrix, e.g. for each -Python version defined in file ``noxconfig.py``: - -.. code-block:: python - - @dataclass(frozen=True) - class Config: - python_versions = ["3.9", "3.10", "3.11", "3.12", "3.13"] - -The GitHub workflows of your project can: - -* Use a build matrix, e.g. using different Python versions as shown above -* Define multiple test sessions, e.g. for distinguishing fast vs. slow or expensive tests. - -The PTB combines the coverage data of all test sessions of the Python -version named first in attribute ``python_versions`` of class ``Config``. diff --git a/doc/user_guide/features/index.rst b/doc/user_guide/features/index.rst index 6bdea16f0..34a6df993 100644 --- a/doc/user_guide/features/index.rst +++ b/doc/user_guide/features/index.rst @@ -6,7 +6,7 @@ Features .. toctree:: :maxdepth: 2 - collecting_metrics + metrics/collecting_metrics creating_a_release documentation/index diff --git a/doc/user_guide/features/metrics/collecting_metrics.rst b/doc/user_guide/features/metrics/collecting_metrics.rst new file mode 100644 index 000000000..b133a8ffa --- /dev/null +++ b/doc/user_guide/features/metrics/collecting_metrics.rst @@ -0,0 +1,82 @@ +Collecting metrics +================== + +.. toctree:: + :maxdepth: 2 + + project_report + sonar + +.. _generated_metrics: + +Generated metrics ++++++++++++++++++ + +The PTB allows you to collect various metrics on the quality of your project +regarding Coverage, Security, and Static Code Analysis. + +For each metric, there is a dedicated nox session, generating one or multiple +files and based on a selected external Python tool. + ++------------------------------------+-----------------------------+--------------+ +| Nox session | Generated files | Based on | ++====================================+=============================+==============+ +| ``lint:code`` | ``lint.txt``, ``lint.json`` | ``pylint`` | ++------------------------------------+-----------------------------+--------------+ +| ``lint:security`` | ``.security.json`` | ``bandit`` | ++------------------------------------+-----------------------------+--------------+ +| ``test:unit -- --coverage`` | ``.coverage`` | ``coverage`` | ++------------------------------------+-----------------------------+--------------+ +| ``test:integration -- --coverage`` | ``.coverage`` | ``coverage`` | ++------------------------------------+-----------------------------+--------------+ + +These metrics are computed for each element in your build matrix, e.g. for each +Python version defined in the file ``noxconfig.py``: + +.. code-block:: python + + @dataclass(frozen=True) + class Config: + python_versions = ["3.9", "3.10", "3.11", "3.12", "3.13"] + +The GitHub workflows of your project can: + +* Use a build matrix, e.g. using different Python versions as shown above +* Define multiple test sessions, e.g. for distinguishing fast vs. slow or expensive tests. + + +Reporting metrics ++++++++++++++++++ + +Currently, the PTB offers two methods to aggregate the :ref:`generated_metrics` +into a report: + +#. the nox session ``project:report`` + This is an Exasol-specific summarization tool. For more information, see :ref:`project_report`. + +#. SonarQube analysis + This summarization tool feeds into a feature-rich UI provided by + `Sonar `__. For further + details, see :ref:`sonarqube_analysis` + +Both of these reporting options require that the generated files from the :ref:`generated_metrics` +are existing and in the expected formats. As there are metrics for different Python +versions, the PTB uses only the metrics associated with the Python version named first +in the attribute ``python_versions`` of class ``Config`` to the reporting metrics tools. + +To perform this validation, there are two nox sessions. Due to the direct +dependence on the nox session ``project:report`` and SonarQube Analysis on the +aforementioned nox sessions, all of these are executed in succession in the CI's ``report.yml``. + ++--------------------------+----------------------------------------------------------+ +| Nox session | Actions | ++==========================+==========================================================+ +| ``artifacts:copy`` | * Combines coverage artifacts from various test sources | +| | (unit, integration ...) | +| | * Copies downloaded artifacts to their parent directory | ++--------------------------+----------------------------------------------------------+ +| ``artifacts:validate`` | * Verifies that the ``.lint.json``, ``.lint.txt``, | +| | ``.security.json``, and ``.coverage`` are present | +| | * Checks that each file contains the expected attributes | +| | for that file type | ++--------------------------+----------------------------------------------------------+ diff --git a/doc/user_guide/features/metrics/project_report.rst b/doc/user_guide/features/metrics/project_report.rst new file mode 100644 index 000000000..6500b2e07 --- /dev/null +++ b/doc/user_guide/features/metrics/project_report.rst @@ -0,0 +1,48 @@ +.. _project_report: + +``project:report`` +================== +After collecting the metrics described in by :ref:`generated_metrics`, you can use the +nox session ``project:report`` to create a report using one of the following formats: + +* JSON + This format is usually used in the CI. The created JSON file is uploaded as an + artifact, which can be downloaded and aggregated for multiple projects + (see :ref:`metrics_schema`). +* markdown + This is displayed in the GitHub Action's summary for a given CI run. Displaying + this content per CI run gives the developer immediate feedback as to how the code + quality has changed between code modifications. + + +.. _metrics_schema: + +Metrics schema +++++++++++++++ +The PTB supports the uniform json schema for metrics used by all projects +of the Exasol Product Integration Team: + +* `Exasol Schemas`_ +* `Metrics Schema`_ +* `Metrics Schema Project`_ + +The Integration team runs regular aggregation of the metrics from all projects into a centralized data sink +as decribed in the company Wiki. + +Development +----------- + +If the metrics schema needs to be updated, the `Metrics Schema Project`_ provides a +convenient way (via a Pydantic model) to update and generate an updated schema for the +metrics. + +.. note:: + + The updated version needs to be first integrated into the `Exasol Schemas Project`_. + + +.. _Exasol Schemas: https://schemas.exasol.com +.. _Exasol Schemas Project: https://github.com/exasol/schemas +.. _Metrics Schema: https://schemas.exasol.com/project-metrics-0.2.0.html +.. _metrics.py: https://github.com/exasol/python-toolbox/blob/main/exasol/toolbox/metrics.py +.. _Metrics Schema Project: https://github.com/exasol/python-toolbox/tree/main/metrics-schema diff --git a/doc/user_guide/features/metrics/sonar.rst b/doc/user_guide/features/metrics/sonar.rst new file mode 100644 index 000000000..a48e4a337 --- /dev/null +++ b/doc/user_guide/features/metrics/sonar.rst @@ -0,0 +1,105 @@ +.. _sonarqube_analysis: + +SonarQube analysis +================== + +The PTB supports using `SonarQube Cloud `__ +to analyze, visualize, & track linting, security, & coverage. All of our Python projects +should be evaluated against the `Exasol Way`_ and subscribe to the +`Clean as You Code `__ +methodology. If code modified in a PR does not satisfy the aforementioned criteria, the +SonarQube analysis fails. + +The PTB includes instructions to set up a GitHub bot to display the results of the +Sonar analysis in your pull requests as a stylized comment and workflow result. +Section :ref:`configuration` gives instructions for public and private repositories. + +.. _configuration: + +Configuration ++++++++++++++ + +.. _configure_sonar_public_project: + +**Public** GitHub repository +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +In GitHub +""""""""" +A GitHub Admin will need to: + +#. Inherit organization secret 'SONAR_TOKEN' +#. Activate the `SonarQubeCloud App `__ +#. **Post-merge**: update the branch protections to include SonarQube analysis. + + * This should only be done when tests exist for the project, & that the project is + at a state in which enforced code coverage would not be a burden. If you do + not enact branch protections, it is recommended to create an issue to do so later. + +In Sonar +"""""""" +#. Create a project on `SonarCloud `__ + + * Project key should follow this pattern, e.g. ``com.exasol:python-toolbox`` + * To alter the project further, you will need the help of a SonarQube Admin. + +In the code +""""""""""" +#. Specify in the ``noxconfig.py`` the relative path to the project's source code in ``Config.source`` + .. code-block:: python + + source: Path = Path("exasol/") +#. Add the following to the project's file ``pyproject.toml`` + .. code-block:: toml + + [tool.sonar] + projectKey = "" + hostUrl = "https://sonarcloud.io" + organization = "exasol" + exclusions = "/version.py,//*" + +.. _configure_sonar_private_project: + +**Private** GitHub repository +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. note:: + As of 2025-07-29, these instructions have not been used. Thus, they should be + scrutinized and refined when they are used to configure a private repository. + +In GitHub +""""""""" +A GitHub Admin will need to: + +#. Add the individual 'PRIVATE_SONAR_TOKEN' to the 'Organization secrets' +#. Activate the `exasonarqubeprchecks App `__ +#. **Post-merge**: update the branch protections to include SonarQube analysis. + + * This should only be done when tests exist for the project, & that the project is + at a state in which enforced code coverage would not be a burden. If you do + not enact branch protections, it is recommended to create an issue to do so later. + +In Sonar +"""""""" +An IT Admin will need to: + +#. Create a project on https://sonar.exasol.com + + * Project key should follow this pattern, e.g. ``com.exasol:python-toolbox`` + + +In the code +""""""""""" +#. Specify in the ``noxconfig.py`` the relative path to the project's source code in ``Config.source`` + .. code-block:: python + + source: Path = Path("exasol/") + +#. Add the following to the project's file ``pyproject.toml`` + .. code-block:: toml + + [tool.sonar] + projectKey = "com.exasol:" + hostUrl = "https://sonar.exasol.com" + organization = "exasol" + exclusions = "/version.py,//*" + +.. _Exasol Way: https://sonarcloud.io/organizations/exasol/quality_gates/show/AXxvLH-3BdtLlpiYmZhh diff --git a/doc/user_guide/getting_started.rst b/doc/user_guide/getting_started.rst index 0766a8646..78eda86ac 100644 --- a/doc/user_guide/getting_started.rst +++ b/doc/user_guide/getting_started.rst @@ -166,67 +166,13 @@ forward, and you just can use the example ``noxfile.py`` below. See :ref:`documentation_configuration` for the required steps. -7. Set up for Sonar -+++++++++++++++++++ -PTB supports using SonarQube Cloud to analyze, visualize, & track linting, security, & -coverage. All of our Python projects are evaluated against the -`Exasol Way `__ -and subscribe to the -`Clean as You Code `__ -methodology, which means that SonarQube analysis will fail and, if its included in the branch protections, block a PR -if code modified in that PR does not meet the standards of the Exasol Way. - -In order to set up Sonar, you will need to perform the following instructions. - -For a **public** project -^^^^^^^^^^^^^^^^^^^^^^^^ -1. Specify in the `noxconfig.py` the relative path to the project's source code in `Config.source` - .. code-block:: python - - source: Path = Path("exasol/") -2. Add the 'SONAR_TOKEN' to the 'Organization secrets' in GitHub (this requires a person being a GitHub organization owner) -3. Activate the `SonarQubeCloud App `_ -4. Create a project on SonarCloud -5. Add the following information to the project's file `pyproject.toml` - .. code-block:: toml - - [tool.sonar] - projectKey = "com.exasol:" - hostUrl = "https://sonarcloud.io" - organization = "exasol" - exclusions = "/version.py,//*" -6. Post-merge, update the branch protections to include SonarQube analysis - - * This should only be done when tests exist for the project, & that the project is - at a state in which enforced code coverage would not be a burden. For new projects, - we recommend creating an issue to add the SonarQube analysis to the branch protections - at a later point. In such scenarios, SonarQube analysis will still report its analysis - results to the PR, but it will not prevent the PR from being merged. - -For a **private** project -^^^^^^^^^^^^^^^^^^^^^^^^^ -1. Specify in the `noxconfig.py` the relative path to the project's source code in `Config.source` - .. code-block:: python - - source: Path = Path("exasol/") -2. Add the 'PRIVATE_SONAR_TOKEN' to the 'Organization secrets' in GitHub (this requires a person being a GitHub organization owner) -3. Activate the `exasonarqubeprchecks App `_ -4. Create a project on https://sonar.exasol.com -5. Add the following information to the project's file `pyproject.toml` - .. code-block:: toml - - [tool.sonar] - projectKey = "com.exasol:" - hostUrl = "https://sonar.exasol.com" - organization = "exasol" - exclusions = "/version.py,//*" -6. Post-merge, update the branch protections to include SonarQube analysis from exasonarqubeprchecks - - * This should only be done when tests exist for the project, & that the project is - at a state in which enforced code coverage would not be a burden. For new projects, - we recommend creating an issue to add the SonarQube analysis to the branch protections - at a later point. In such scenarios, SonarQube analysis will still report its analysis - results to the PR, but it will not prevent the PR from being merged. +7. Set up Sonar ++++++++++++++++ + +Look at the configuration of Sonar for a: + +* :ref:`configure_sonar_public_project` +* :ref:`configure_sonar_private_project` 8. Go 🥜 +++++++++++++