Skip to content

Commit 39cadfd

Browse files
jsilvelaNiccoloFei
jsilvela
and
NiccoloFei
authored
Improve how CIclops is consumed by calling workflows (#5)
- use GitHub env vars inside the action, rather than generate files - compute the overflow logic inside the action - leverage GitHub outputs for alerts and overflow - update the documents and manifests - anonymize test JSONs Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com> Co-authored-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
1 parent d4314a5 commit 39cadfd

File tree

20 files changed

+176
-186
lines changed

20 files changed

+176
-186
lines changed

‎.github/workflows/overflow-test.yaml‎

Lines changed: 9 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
name: Overflow Test
2+
13
on: [push]
24

35
defaults:
@@ -8,60 +10,21 @@ defaults:
810
jobs:
911
overflow_test:
1012
runs-on: ubuntu-latest
11-
name: Test CIclops with summary overflow
13+
name: Overflow Test
1214
steps:
1315
- uses: actions/checkout@v3
1416

1517
- name: Generate Test Summary
18+
id: generate-summary
1619
uses: ./
1720
with:
1821
artifact_directory: example-artifacts/
19-
output_file: test-summary.md
20-
short_file: short.md
21-
alerts_file: alerts.txt
22-
# NOTE: it does not seem possible to pass $GITHUB_STEP_SUMMARY as a
23-
# regular file to the underlying script. Hence, we create a file and
24-
# in a later step write with >> $GITHUB_STEP_SUMMARY
25-
26-
- name: Create local file that is bigger than GH limit
27-
run: |
28-
dd if=/dev/zero of=big-test-summary.md bs=1M count=2
29-
30-
- name: Check full summary fits within GH limit
31-
# $GITHUB_STEP_SUMMARY will reject content over 1024 bytes
32-
# on exceeding, the workflow WILL FAIL and still count as success()
33-
# With this step, we do proper error flow, and fail if the limit would be
34-
# exceeded.
35-
id: check-overflow
36-
run: |
37-
size=$(stat -c '%s' big-test-summary.md)
38-
if [ "$size" -gt 1024 ]; then
39-
echo "overflow=true" >> $GITHUB_OUTPUT
40-
fi
41-
# Here we force the "big test summary" to overflow GH limits, and we
42-
# create an output that further steps can leverage: steps.check-overflow.outputs.overflow
22+
limit_summary: 10
4323

44-
- name: If the full summary fits within GH limits, use it
45-
# This step should be skipped, we expect
46-
if: ${{!steps.check-overflow.outputs.overflow}}
47-
run: |
48-
cat big-test-summary.md >> $GITHUB_STEP_SUMMARY
49-
50-
- name: If the full summary is too big, use short version
51-
if: ${{steps.check-overflow.outputs.overflow}}
52-
run: |
53-
cat short.md >> $GITHUB_STEP_SUMMARY
54-
55-
- name: If full summary is too big, archive it
56-
if: ${{steps.check-overflow.outputs.overflow}}
24+
- name: If there is an overflow summary, archive it
25+
if: ${{steps.generate-summary.outputs.Overflow}}
5726
uses: actions/upload-artifact@v3
5827
with:
59-
name: test-summary.md
60-
path: test-summary.md
28+
name: ${{steps.generate-summary.outputs.Overflow}}
29+
path: ${{steps.generate-summary.outputs.Overflow}}
6130
retention-days: 7
62-
63-
- name: Create slack body with alerts
64-
run: |
65-
echo 'slack-message<<EOF' >> $GITHUB_OUTPUT
66-
cat alerts.txt >> $GITHUB_OUTPUT
67-
echo 'EOF' >> $GITHUB_OUTPUT

‎.github/workflows/test.yaml‎

Lines changed: 8 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,20 @@
1+
name: Smoke Test
2+
13
on: [push]
24

35
jobs:
4-
plain_test:
6+
smoke_test:
57
runs-on: ubuntu-latest
6-
name: Test CIclops with normal summary
8+
name: Smoke Test
79
steps:
810
- uses: actions/checkout@v3
911

1012
- name: Generate Test Summary
13+
id: generate-summary
1114
uses: ./
1215
with:
1316
artifact_directory: example-artifacts/
14-
output_file: test-summary.md
15-
short_file: short.md
16-
alerts_file: alerts.txt
17-
# NOTE: it does not seem possible to pass $GITHUB_STEP_SUMMARY as a
18-
# regular file to the underlying script. Hence, we create a file and
19-
# in a later step write with >> $GITHUB_STEP_SUMMARY
20-
21-
- name: Create GitHub Job Summary from report
22-
run: |
23-
cat test-summary.md >> $GITHUB_STEP_SUMMARY
2417

25-
- name: Create slack body with alerts
26-
run: |
27-
echo 'slack-message<<EOF' >> $GITHUB_OUTPUT
28-
cat alerts.txt >> $GITHUB_OUTPUT
29-
echo 'EOF' >> $GITHUB_OUTPUT
18+
- name: If there are alerts, echo them
19+
if: ${{steps.generate-summary.outputs.alerts}}
20+
run: echo "${{steps.generate-summary.outputs.alerts}}"

‎.github/workflows/unit-test.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
1+
name: Unit Test
2+
13
on: [push]
24

35
jobs:
4-
plain_test:
6+
unit_test:
57
runs-on: ubuntu-latest
68
name: Unit test
79
steps:

‎DEVELOPERS_DEVELOPERS_DEVELOPERS.md‎

Lines changed: 32 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,12 @@ where you can see some JSON artifacts in the expected format. For example:
1212
python summarize_test_results.py --dir example-artifacts
1313
```
1414

15+
or
16+
17+
``` shell
18+
GITHUB_STEP_SUMMARY=out.md python summarize_test_results.py --dir example-artifacts
19+
```
20+
1521
You can build the container with
1622

1723
``` shell
@@ -22,22 +28,41 @@ To get a better idea of how *ciclops* will work when used in GitHub workflows,
2228
it is useful to run locally with `act`. See
2329
[*act* homepage](https://github.com/nektos/act) for reference.
2430

25-
In the `.github/workflows` directory in this repo, you will find a test YAML
26-
workflow file you can run with `act`.
31+
In the `.github/workflows` directory in this repo, you will find test YAML
32+
workflows you can run with `act`.
2733

2834
**WARNING**: to test with `act`, take care to use the `-b` option to **bind**
2935
the working directory to the Docker container. The default behavior of copying
3036
will not work properly (at least at the time of testing this, September 2022.)
37+
Also, make sure you are using at least the *Medium* size Docker images given
38+
that the `-slim` ones don't support Python.
39+
40+
The following instruction will execute all available workflows:
41+
42+
``` shell
43+
act -b
44+
```
3145

32-
`act` does not have direct support for the GitHub Job Summaries.
33-
See [`act` issue for GH job summary](https://github.com/nektos/act/issues/1187).
34-
As a workaround, we can use the `--env` option. Example:
46+
You can see the list of all possible jobs with `act -l`.
47+
If you want to specify a particular job in a particular workflow, do e.g.:
3548

3649
``` shell
37-
act -b --env GITHUB_STEP_SUMMARY='github-summary.md'
50+
act -b -j smoke_test -W .github/workflows/test.yaml
3851
```
3952

40-
Running this should create a file `github-summary.md` with the test summary.
53+
**NOTE**: `act` will provide a testing environment close to that of GitHub.
54+
In particular, the variables GITHUB_STEP_SUMMARY and GITHUB_OUTPUT are
55+
populated, and will be available to the Python script within the Docker image.
56+
57+
**HINT**: some workflows, like `overflow-test.yaml`, will try to upload a
58+
CIclops output file as an artifact by using `actions/upload-artifact`.
59+
In such cases you may have to specify a path to your artifact server by
60+
using the `--artifact-server-path` option. For example:
61+
62+
``` shell
63+
mkdir /tmp/artifacts
64+
act -b -j overflow_test -W .github/workflows/overflow-test.yaml --artifact-server-path /tmp/artifacts
65+
```
4166

4267
## Unit tests
4368

‎Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,4 @@ RUN pip install --no-cache-dir -r requirements.txt
77
COPY . .
88

99
ENTRYPOINT [ "python", "/summarize_test_results.py"]
10-
CMD ["--dir", "./test-artifacts", "--out", "", "--short", "", "--alerts", ""]
10+
CMD ["--dir", "./test-artifacts"]

‎README.md‎

Lines changed: 39 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,23 @@ watch over Continuous Integration pipelines for all eternity.
1313

1414
- `artifact_directory`: **Required** directory holding the
1515
JSON artifacts representing tests.
16+
1617
- `output_file`: **Optional** the file where the markdown report will be
17-
written. If omitted, the report will be written to Standard-Out.
18+
written. If omitted, the report will be written to the `GITHUB_STEP_SUMMARY`
19+
environment variable used by GitHub to display job summaries.
1820

1921
## Outputs
2022

21-
No outputs in the GitHub Actions sense. It does write a test summary in Markdown
22-
format, either to a file or to Stdout depending on the `output_file` input.
23+
Two outputs might be produced:
24+
25+
- `alerts`: this will contain stand-alone text with systematic failures
26+
detected by CIclops. It is meant to enable further steps in the calling
27+
workflow, for example sending to Slack for monitoring of the CI/CD health.
28+
29+
- `Overflow`: this will contain the name of a file with the full Summary. This
30+
file will be created if the summary exceeded the limit of 1024K imposed by
31+
GitHub for job summaries. A shorter summary will be output, and the filename
32+
indicated by `Overflow` may be used by the calling workflow for Archival.
2333

2434
## Usage
2535

@@ -71,10 +81,6 @@ For example:
7181
uses: cloudnative-pg/ciclops@main
7282
with:
7383
artifact_directory: test-artifacts/data
74-
output_file: test-summary.md
75-
76-
- name: Create GitHub Job Summary from the file
77-
run: cat test-summary.md >> $GITHUB_STEP_SUMMARY
7884
```
7985
8086
## Advanced Usage
@@ -83,73 +89,56 @@ There are two advanced cases we want to call attention to:
8389
8490
1. Summary overflow. \
8591
The `GITHUB_STEP_SUMMARY` variable set to receive the CIclops summary will
86-
overflow if the summary is bigger than 1024 bytes. To work around this,
87-
CIclops can be directed to create a short summary on top of the "normal"
88-
summary. The calling workflow can verify if the full summary is too big, and
89-
if so, can display the short summary, and perhaps Archive the full summary.
90-
91-
2. Slackops \
92+
overflow if the summary is bigger than 1024K. To work around this, in case of
93+
overflow CIclops will create a short version of the summary and overwrite
94+
GITHUB_STEP_SUMMARY, and in addition it will write the full summary to a file
95+
that a calling workflow can Archive.
96+
The name of the full file is sent a GitHub action *output* variable
97+
called `Overflow`.
98+
99+
2. Monitoring with chatops \
92100
CIclops will create a series of alerts when systematic failures are detected.
93101
By "systematic", we mean cases such as:
94102

95103
- all test combinations have failed
96104
- all combinations fail for a given test
97105
- all tests fail for a given version of Postgres
98106

99-
The alerts are included in the summary generated by CIclops, but it is also
100-
possible to direct CIclops to put the alerts only in an output file.
101-
This file can then be sent via Slack message to alert DevOps teams.
107+
The alerts are included in the summary generated by CIclops. In addition,
108+
CIclops will send the alerts alone as a GitHub action *output* variable
109+
called `alerts`.
110+
The alerts can then be sent via Slack message to alert DevOps teams.
102111

103112
The following snippet shows how to use these features:
104113

105114
``` yaml
115+
…
116+
…
117+
steps:
118+
…
119+
…
106120
- name: Generate Test Summary
121+
id: generate-summary
107122
uses: cloudnative-pg/ciclops@main
108123
with:
109124
artifact_directory: test-artifacts/data
110-
output_file: test-summary.md
111-
short_file: short.md
112-
alerts_file: alerts.txt
113125
114-
- name: Check full summary fits within GH limit
115-
id: check-overflow
116-
run: |
117-
size=$(stat -c '%s' test-summary.md)
118-
if [ "$size" -gt 1024 ]; then
119-
echo "overflow=true" >> $GITHUB_OUTPUT
120-
fi
121-
122-
- name: If the full summary would not overflow, use it
123-
if: ${{!steps.check-overflow.outputs.overflow}}
124-
run: |
125-
cat test-summary.md >> $GITHUB_STEP_SUMMARY
126-
127-
- name: If the full summary is too big, use short version
128-
if: ${{steps.check-overflow.outputs.overflow}}
129-
run: |
130-
cat short.md >> $GITHUB_STEP_SUMMARY
131-
132-
- name: If full summary is too big, archive it
133-
if: ${{steps.check-overflow.outputs.overflow}}
126+
- name: If there is an overflow summary, archive it
127+
if: ${{steps.generate-summary.outputs.Overflow}}
134128
uses: actions/upload-artifact@v3
135129
with:
136-
name: test-summary.md
137-
path: test-summary.md
130+
name: ${{steps.generate-summary.outputs.Overflow}}
131+
path: ${{steps.generate-summary.outputs.Overflow}}
138132
retention-days: 7
139133
140-
- name: Create Slack body with alerts
141-
id: alerts
142-
run: |
143-
echo 'slack-message<<EOF' >> $GITHUB_OUTPUT
144-
cat alerts.txt >> $GITHUB_OUTPUT
145-
echo 'EOF' >> $GITHUB_OUTPUT
146-
147-
- name: Send Slack Notification
134+
- name: If there are alerts, send them over Slack
135+
if: ${{steps.generate-summary.outputs.alerts}}
148136
uses: rtCamp/action-slack-notify@v2
149137
env:
150138
SLACK_USERNAME: cnpg-bot
151139
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
152-
SLACK_MESSAGE: ${{ steps.alerts.outputs.slack-message }}
140+
SLACK_MESSAGE: ${{steps.generate-summary.outputs.alerts}}
141+
…
153142
```
154143

155144
## Origin

‎action.yaml‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,16 @@ inputs:
1414
output_file:
1515
description: "file where the markdown report should be written"
1616
required: false
17-
short_file:
18-
description: "file where the abridged markdown report should be written"
19-
required: false
20-
alerts_file:
21-
description: "file where any alerts found should be written"
17+
limit_summary:
18+
description: "limit in bytes that can be sent to GITHUB_STEP_SUMMARY"
2219
required: false
20+
# GitHub imposes a 1024K limit for GITHUB_STEP_SUMMARY
21+
default: 1024000
22+
outputs:
23+
alerts:
24+
description: 'Any systematic failures found by CIclops'
25+
Overflow:
26+
description: 'The name of the file where the full report was written, on oveflow'
2327
runs:
2428
using: "docker"
2529
image: "Dockerfile"
@@ -28,7 +32,5 @@ runs:
2832
- "./${{ inputs.artifact_directory }}"
2933
- "--out"
3034
- "${{ inputs.output_file }}"
31-
- "--short"
32-
- "${{ inputs.short_file }}"
33-
- "--alerts"
34-
- "${{ inputs.alerts_file }}"
35+
- "--limit"
36+
- ${{ inputs.limit_summary }}

‎example-artifacts/local-v1.21.14-PGD-PostgreSQL-12.13-5.0.0-0.0git256.g2a1b38b.1.dev-1_86b346ab9958ad86dc1829a4b9f509415ce8da7d997e5ea40bb24af8.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
"state": "failed",
44
"start_time": "2023-01-25T10:26:03.771385171Z",
55
"end_time": "2023-01-25T10:38:44.769651724Z",
6-
"error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc0003c4c20>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1",
7-
"error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go",
6+
"error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc0003c4c20>: {\n s: \"Assertion in callback at /home/runner/work/my-repo/my-project/my-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/my-repo/my-project/my-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1",
7+
"error_file": "/home/runner/work/my-repo/my-project/my-operator/tests/e2e/asserts_test.go",
88
"error_line": 268,
99
"platform": "local",
1010
"postgres_kind": "PostgreSQL",
@@ -13,6 +13,6 @@
1313
"postgres_version": "12.13-5.0.0-0.0git256.g2a1b38b.1.dev-1",
1414
"k8s_version": "v1.21.14",
1515
"workflow_id": 4004501040,
16-
"repo": "EnterpriseDB/pg4k-pgd",
16+
"repo": "my-org/my-rep",
1717
"branch": "dev/cnp-3285-2"
1818
}

‎example-artifacts/local-v1.21.14-PGD-PostgreSQL-13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1_86b346ab9958ad86dc1829a4b9f509415ce8da7d997e5ea40bb24af8.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
"state": "failed",
44
"start_time": "2023-01-25T11:08:20.179946291Z",
55
"end_time": "2023-01-25T11:20:54.376679592Z",
6-
"error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc000a41e40>: {\n s: \"Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1",
7-
"error_file": "/home/runner/work/pg4k-pgd/pg4k-pgd/pgd-operator/tests/e2e/asserts_test.go",
6+
"error": "Timed out after 600.001s.\nExpected success, but got an error:\n <*errors.errorString | 0xc000a41e40>: {\n s: \"Assertion in callback at /home/runner/work/my-repo/my-project/my-operator/tests/e2e/asserts_test.go:266 failed:\\nExpected\\n <int32>: 0\\nto equal\\n <int32>: 1\",\n }\n Assertion in callback at /home/runner/work/my-repo/my-project/my-operator/tests/e2e/asserts_test.go:266 failed:\n Expected\n <int32>: 0\n to equal\n <int32>: 1",
7+
"error_file": "/home/runner/work/my-repo/my-project/my-operator/tests/e2e/asserts_test.go",
88
"error_line": 268,
99
"platform": "local",
1010
"postgres_kind": "PostgreSQL",
@@ -13,6 +13,6 @@
1313
"postgres_version": "13.9-5.0.0-0.0git256.g2a1b38b.1.dev-1",
1414
"k8s_version": "v1.21.14",
1515
"workflow_id": 4004501040,
16-
"repo": "EnterpriseDB/pg4k-pgd",
16+
"repo": "my-org/my-rep",
1717
"branch": "dev/cnp-3285-2"
1818
}

0 commit comments

Comments
 (0)