feat: create a test thermometer (#10)

- Introduce a "thermometer" both for the GH summary, and as an
   output that can be used in chatops
- Stop overcounting kubernetes versions and failure rates in them
- Cap the number of alerts shown in chatops (max 2 per metric)
- Update the GH actions dependencies
Closes #9

Signed-off-by: Jaime Silvela <[email protected]>
Signed-off-by: Niccolò Fei <[email protected]>
Signed-off-by: John Long <[email protected]>
Co-authored-by: Niccolò Fei <[email protected]>
Co-authored-by: John Long <[email protected]>

Author: 3 people

.github/workflows/overflow-test.yaml

@@ -12,7 +12,7 @@ jobs:
     runs-on: ubuntu-latest
     name: Overflow Test
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Generate Test Summary
         id: generate-summary
@@ -23,7 +23,7 @@ jobs:
 
       - name: If there is an overflow summary, archive it
         if: ${{steps.generate-summary.outputs.Overflow}}
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: ${{steps.generate-summary.outputs.Overflow}}
           path: ${{steps.generate-summary.outputs.Overflow}}

.github/workflows/test.yaml

@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     name: Smoke Test
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Generate Test Summary
         id: generate-summary
@@ -18,3 +18,6 @@ jobs:
       - name: If there are alerts, echo them
         if: ${{steps.generate-summary.outputs.alerts}}
         run: echo "${{steps.generate-summary.outputs.alerts}}"
+
+      - name: Echo the thermometer
+        run: echo "${{steps.generate-summary.outputs.thermometer}}"

.github/workflows/unit-test.yaml

@@ -7,10 +7,10 @@ jobs:
     runs-on: ubuntu-latest
     name: Unit test
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
 
     - name: Set up Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
         python-version: '3.x'
 

DEVELOPERS_DEVELOPERS_DEVELOPERS.md

@@ -1,8 +1,25 @@
-# Building and testing locally
+# Building, testing, releasing
 
 The `ciclops` GitHub Action runs using a Docker container that encapsulates the
 Python script that does the CI test analysis.
 
+## Releasing
+
+We recommend that users of Ciclops use released versions rather than `main`.
+For testing, it  may be convenient to [use a full SHA](#testing-within-a-calling-github-workflow).
+
+The procedure for cutting a release:
+
+1. Decide on the version number (following semVer)
+1. Update the [Release notes file](ReleaseNotes.md), following the convention
+  in the file, i.e. the version number included in the section, and the release
+  date in the first line
+1. Review and merge the release notes, and create and push a new tag with the
+  desired version number
+1. Cut a new release in [GitHub](https://github.com/cloudnative-pg/ciclops/releases/new),
+  choosing the recent tag, and pasting the relevant content from the
+  Release Notes file (no need for the release date line).
+
 ## Developing and testing
 
 You can test directly with the Python code on the `example-artifacts` directory,
@@ -72,6 +89,22 @@ CIclops has the beginning of a unit test suite. You can run it with:
 python3 -m unittest
 ```
 
+## Testing within a calling GitHub workflow
+
+Even with unit tests and local tests, it's good to try Ciclops code out from a
+client workflow. We can use a full length commit SHA to test out changes,
+before cutting out a new release.
+See the [GitHub document on using third party actions](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-third-party-actions).
+
+Example:
+``` yaml
+      - name: Compute the E2E test summary
+        id: generate-summary
+        uses: cloudnative-pg/ciclops@<FULL_LENGTH_SHA>
+        with:
+          artifact_directory: test-artifacts/da
+```
+
 ## How it works
 
 The files in this repository are needed for the Dockerfile to build and run, of

README.md

@@ -20,7 +20,11 @@ watch over Continuous Integration pipelines for all eternity.
 
 ## Outputs
 
-Two outputs might be produced:
+Up to three outputs might be produced:
+
+- `thermometer`: this will contain stand-alone text with a color-coded list
+  of test metrics that can serve as an overview of the state of the test suite
+  on CI/CD. This is generated on every execution of Ciclops.
 
 - `alerts`: this will contain stand-alone text with systematic failures
   detected by CIclops. It is meant to enable further steps in the calling
@@ -97,7 +101,12 @@ There are two advanced cases we want to call attention to:
   called `Overflow`.
 
 2. Monitoring with chatops \
-  CIclops will create a series of alerts when systematic failures are detected.
+  Ciclops will generate a "thermometer" on every execution, offering a
+  color-coded overview of the test health. This thermometer is included in
+  the GitHub summary, and in addition, is exported as an output in plain
+  text, which can be sent via chatops.
+  In addition, Ciclops will create a series of alerts when systematic failures
+  are detected.
   By "systematic", we mean cases such as:
 
     - all test combinations have failed
@@ -131,6 +140,13 @@ The following snippet shows how to use these features:
           path: ${{steps.generate-summary.outputs.Overflow}}
           retention-days: 7
 
+      - name: Get a slack message with the Ciclops thermometer
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_USERNAME: cnpg-bot
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
+          SLACK_MESSAGE: ${{steps.generate-summary.outputs.thermometer}}
+
       - name: If there are alerts, send them over Slack
         if: ${{steps.generate-summary.outputs.alerts}}
         uses: rtCamp/action-slack-notify@v2

action.yaml

@@ -22,6 +22,8 @@ inputs:
 outputs:
   alerts:
     description: 'Any systematic failures found by CIclops'
+  thermometer:
+    description: 'A color-coded health meter'
   Overflow:
     description: 'The name of the file where the full report was written, on oveflow'
 runs:

example-artifacts/id1_0b185c51a60964ecab5bb7d97458ca95fd421f325f3896ed239d5d3f.json

@@ -10,7 +10,7 @@
     "postgres_kind": "PostgreSQL",
     "matrix_id": "id1",
     "postgres_version": "11.1",
-    "k8s_version": "22",
+    "k8s_version": "1.22",
     "workflow_id": 12,
     "repo": "my-repo",
     "branch": "my-branch"

example-artifacts/id1_4902843ff6a60bc4fdb76000698c46de8e7f9763a1a0fe63f70fd5f8.json

@@ -10,7 +10,7 @@
     "postgres_kind": "PostgreSQL",
     "matrix_id": "id1",
     "postgres_version": "11.1",
-    "k8s_version": "22",
+    "k8s_version": "1.22",
     "workflow_id": 12,
     "repo": "my-repo",
     "branch": "my-branch"

example-artifacts/id1_9891d0d1caa1ec8fd0adfe622e341f84dd3d1df1cffee843e1fb84a0.json

@@ -10,7 +10,7 @@
     "postgres_kind": "PostgreSQL",
     "matrix_id": "id1",
     "postgres_version": "11.1",
-    "k8s_version": "22",
+    "k8s_version": "1.22",
     "workflow_id": 12,
     "repo": "my-repo",
     "branch": "my-branch"

few-artifacts/id1_0b185c51a60964ecab5bb7d97458ca95fd421f325f3896ed239d5d3f.json

@@ -10,7 +10,7 @@
     "postgres_kind": "PostgreSQL",
     "matrix_id": "id1",
     "postgres_version": "11.1",
-    "k8s_version": "22",
+    "k8s_version": "1.22",
     "workflow_id": 12,
     "repo": "my-repo",
     "branch": "my-branch"