Skip to content

Commit d4314a5

Browse files
jsilvelaNiccoloFei
andauthored
feat: compute alerts on systematic failure (#4)
Stop showing the ignoreFail as a code error. Create a short summary for cases where the full summary would be truncated by GH. Compute an Alerts section and make it available standalone for use in Slackops. Add unit tests. Signed-off-by: Jaime Silvela <jaime.silvela@enterprisedb.com> Signed-off-by: Tao Li <tao.li@enterprisedb.com> Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com> Co-authored-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
1 parent 8b2bd1b commit d4314a5

File tree

27 files changed

+641
-23
lines changed

27 files changed

+641
-23
lines changed
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
on: [push]
2+
3+
defaults:
4+
run:
5+
# default failure handling for shell scripts in 'run' steps
6+
shell: 'bash -Eeuo pipefail -x {0}'
7+
8+
jobs:
9+
overflow_test:
10+
runs-on: ubuntu-latest
11+
name: Test CIclops with summary overflow
12+
steps:
13+
- uses: actions/checkout@v3
14+
15+
- name: Generate Test Summary
16+
uses: ./
17+
with:
18+
artifact_directory: example-artifacts/
19+
output_file: test-summary.md
20+
short_file: short.md
21+
alerts_file: alerts.txt
22+
# NOTE: it does not seem possible to pass $GITHUB_STEP_SUMMARY as a
23+
# regular file to the underlying script. Hence, we create a file and
24+
# in a later step write with >> $GITHUB_STEP_SUMMARY
25+
26+
- name: Create local file that is bigger than GH limit
27+
run: |
28+
dd if=/dev/zero of=big-test-summary.md bs=1M count=2
29+
30+
- name: Check full summary fits within GH limit
31+
# $GITHUB_STEP_SUMMARY will reject content over 1024 bytes
32+
# on exceeding, the workflow WILL FAIL and still count as success()
33+
# With this step, we do proper error flow, and fail if the limit would be
34+
# exceeded.
35+
id: check-overflow
36+
run: |
37+
size=$(stat -c '%s' big-test-summary.md)
38+
if [ "$size" -gt 1024 ]; then
39+
echo "overflow=true" >> $GITHUB_OUTPUT
40+
fi
41+
# Here we force the "big test summary" to overflow GH limits, and we
42+
# create an output that further steps can leverage: steps.check-overflow.outputs.overflow
43+
44+
- name: If the full summary fits within GH limits, use it
45+
# This step should be skipped, we expect
46+
if: ${{!steps.check-overflow.outputs.overflow}}
47+
run: |
48+
cat big-test-summary.md >> $GITHUB_STEP_SUMMARY
49+
50+
- name: If the full summary is too big, use short version
51+
if: ${{steps.check-overflow.outputs.overflow}}
52+
run: |
53+
cat short.md >> $GITHUB_STEP_SUMMARY
54+
55+
- name: If full summary is too big, archive it
56+
if: ${{steps.check-overflow.outputs.overflow}}
57+
uses: actions/upload-artifact@v3
58+
with:
59+
name: test-summary.md
60+
path: test-summary.md
61+
retention-days: 7
62+
63+
- name: Create slack body with alerts
64+
run: |
65+
echo 'slack-message<<EOF' >> $GITHUB_OUTPUT
66+
cat alerts.txt >> $GITHUB_OUTPUT
67+
echo 'EOF' >> $GITHUB_OUTPUT

.github/workflows/test.yaml

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
on: [push]
22

33
jobs:
4-
custom_test:
4+
plain_test:
55
runs-on: ubuntu-latest
6-
name: Test action with act
6+
name: Test CIclops with normal summary
77
steps:
88
- uses: actions/checkout@v3
99

@@ -12,8 +12,18 @@ jobs:
1212
with:
1313
artifact_directory: example-artifacts/
1414
output_file: test-summary.md
15+
short_file: short.md
16+
alerts_file: alerts.txt
1517
# NOTE: it does not seem possible to pass $GITHUB_STEP_SUMMARY as a
1618
# regular file to the underlying script. Hence, we create a file and
1719
# in a later step write with >> $GITHUB_STEP_SUMMARY
20+
1821
- name: Create GitHub Job Summary from report
19-
run: cat test-summary.md >> $GITHUB_STEP_SUMMARY
22+
run: |
23+
cat test-summary.md >> $GITHUB_STEP_SUMMARY
24+
25+
- name: Create slack body with alerts
26+
run: |
27+
echo 'slack-message<<EOF' >> $GITHUB_OUTPUT
28+
cat alerts.txt >> $GITHUB_OUTPUT
29+
echo 'EOF' >> $GITHUB_OUTPUT

.github/workflows/unit-test.yaml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
on: [push]
2+
3+
jobs:
4+
plain_test:
5+
runs-on: ubuntu-latest
6+
name: Unit test
7+
steps:
8+
- uses: actions/checkout@v3
9+
10+
- name: Set up Python
11+
uses: actions/setup-python@v4
12+
with:
13+
python-version: '3.x'
14+
15+
- name: Install dependencies
16+
run: python -m pip install --upgrade pip prettytable
17+
18+
- name: Run suite
19+
run: python test_summary.py -v

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
# editor and IDE paraphernalia
22
.idea
3+
__pycache__/

DEVELOPERS_DEVELOPERS_DEVELOPERS.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# building and testing locally
1+
# Building and testing locally
22

33
The `ciclops` GitHub Action runs using a Docker container that encapsulates the
44
Python script that does the CI test analysis.
@@ -39,6 +39,14 @@ act -b --env GITHUB_STEP_SUMMARY='github-summary.md'
3939

4040
Running this should create a file `github-summary.md` with the test summary.
4141

42+
## Unit tests
43+
44+
CIclops has the beginning of a unit test suite. You can run it with:
45+
46+
``` sh
47+
python3 -m unittest
48+
```
49+
4250
## How it works
4351

4452
The files in this repository are needed for the Dockerfile to build and run, of
@@ -57,4 +65,4 @@ See [GitHub support for Dockerfile](https://docs.github.com/en/actions/creating-
5765
5866
**NOTE**: the behavior of the `COPY` command in Dockerfiles seems quite
5967
finicky on whether it's done recursively or not. The invocation used,
60-
`COPY . .`, ensured that the copy was recursive.
68+
`COPY . .`, ensures the copy is done recursively.

Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,4 @@ RUN pip install --no-cache-dir -r requirements.txt
77
COPY . .
88

99
ENTRYPOINT [ "python", "/summarize_test_results.py"]
10-
CMD ["--dir", "./test-artifacts", "--out", ""]
10+
CMD ["--dir", "./test-artifacts", "--out", "", "--short", "", "--alerts", ""]

README.md

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,81 @@ For example:
7777
run: cat test-summary.md >> $GITHUB_STEP_SUMMARY
7878
```
7979
80+
## Advanced Usage
81+
82+
There are two advanced cases we want to call attention to:
83+
84+
1. Summary overflow. \
85+
The `GITHUB_STEP_SUMMARY` variable set to receive the CIclops summary will
86+
overflow if the summary is bigger than 1024 bytes. To work around this,
87+
CIclops can be directed to create a short summary on top of the "normal"
88+
summary. The calling workflow can verify if the full summary is too big, and
89+
if so, can display the short summary, and perhaps Archive the full summary.
90+
91+
2. Slackops \
92+
CIclops will create a series of alerts when systematic failures are detected.
93+
By "systematic", we mean cases such as:
94+
95+
- all test combinations have failed
96+
- all combinations fail for a given test
97+
- all tests fail for a given version of Postgres
98+
99+
The alerts are included in the summary generated by CIclops, but it is also
100+
possible to direct CIclops to put the alerts only in an output file.
101+
This file can then be sent via Slack message to alert DevOps teams.
102+
103+
The following snippet shows how to use these features:
104+
105+
``` yaml
106+
- name: Generate Test Summary
107+
uses: cloudnative-pg/ciclops@main
108+
with:
109+
artifact_directory: test-artifacts/data
110+
output_file: test-summary.md
111+
short_file: short.md
112+
alerts_file: alerts.txt
113+
114+
- name: Check full summary fits within GH limit
115+
id: check-overflow
116+
run: |
117+
size=$(stat -c '%s' test-summary.md)
118+
if [ "$size" -gt 1024 ]; then
119+
echo "overflow=true" >> $GITHUB_OUTPUT
120+
fi
121+
122+
- name: If the full summary would not overflow, use it
123+
if: ${{!steps.check-overflow.outputs.overflow}}
124+
run: |
125+
cat test-summary.md >> $GITHUB_STEP_SUMMARY
126+
127+
- name: If the full summary is too big, use short version
128+
if: ${{steps.check-overflow.outputs.overflow}}
129+
run: |
130+
cat short.md >> $GITHUB_STEP_SUMMARY
131+
132+
- name: If full summary is too big, archive it
133+
if: ${{steps.check-overflow.outputs.overflow}}
134+
uses: actions/upload-artifact@v3
135+
with:
136+
name: test-summary.md
137+
path: test-summary.md
138+
retention-days: 7
139+
140+
- name: Create Slack body with alerts
141+
id: alerts
142+
run: |
143+
echo 'slack-message<<EOF' >> $GITHUB_OUTPUT
144+
cat alerts.txt >> $GITHUB_OUTPUT
145+
echo 'EOF' >> $GITHUB_OUTPUT
146+
147+
- name: Send Slack Notification
148+
uses: rtCamp/action-slack-notify@v2
149+
env:
150+
SLACK_USERNAME: cnpg-bot
151+
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
152+
SLACK_MESSAGE: ${{ steps.alerts.outputs.slack-message }}
153+
```
154+
80155
## Origin
81156

82157
At EDB, working on a series of Kubernetes operators for PostgreSQL, we have an

action.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,12 @@ inputs:
1414
output_file:
1515
description: "file where the markdown report should be written"
1616
required: false
17+
short_file:
18+
description: "file where the abridged markdown report should be written"
19+
required: false
20+
alerts_file:
21+
description: "file where any alerts found should be written"
22+
required: false
1723
runs:
1824
using: "docker"
1925
image: "Dockerfile"
@@ -22,3 +28,7 @@ runs:
2228
- "./${{ inputs.artifact_directory }}"
2329
- "--out"
2430
- "${{ inputs.output_file }}"
31+
- "--short"
32+
- "${{ inputs.short_file }}"
33+
- "--alerts"
34+
- "${{ inputs.alerts_file }}"

example-artifacts/id1_0b185c51a60964ecab5bb7d97458ca95fd421f325f3896ed239d5d3f.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,4 @@
1414
"workflow_id": 12,
1515
"repo": "my-repo",
1616
"branch": "my-branch"
17-
}
17+
}

example-artifacts/id1_4902843ff6a60bc4fdb76000698c46de8e7f9763a1a0fe63f70fd5f8.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,4 @@
1414
"workflow_id": 12,
1515
"repo": "my-repo",
1616
"branch": "my-branch"
17-
}
17+
}

0 commit comments

Comments
 (0)