Skip to content

Commit 158efdf

Browse files
authored
docs: improve ncu-ci docs (#453)
* docs: improve ncu-ci docs * Fixup malformed ncu-ci walk <type> section
1 parent be9e3ff commit 158efdf

File tree

1 file changed

+170
-17
lines changed

1 file changed

+170
-17
lines changed

docs/ncu-ci.md

Lines changed: 170 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -30,47 +30,200 @@ Options:
3030
--help Show help [boolean]
3131
```
3232

33-
## Example
33+
### `ncu-ci rate <type>`
3434

35-
Get the CI results of PR 12345 (including latest results of each type of
36-
supported CI) and copy the summaries into clipboard:
35+
`ncu-ci rate <type>` calculate the success rate for CI jobs in the last 100 runs per [CI Health History](https://github.com/nodejs/reliability#ci-health-history), where `<type>` can be either `pr` for `node-test-pull-request` or `commit` for `node-test-commit`.
3736

38-
```
39-
ncu-ci url https://github.com/nodejs/node/pull/12345 --copy
40-
```
37+
Examples:
4138

42-
Get the results of job #12345 of `node-test-pull-request`:
39+
```sh
40+
node on git:master ❯ ncu-ci rate pr
41+
--------------------------------------------------------------------------------
42+
[1/1] Running health
43+
--------------------------------------------------------------------------------
44+
✔ Done
45+
| UTC Time | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
46+
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
47+
| 2020-07-03 16:28 | 1 | 7 | 34 | 10 | 48 | 7.87% |
48+
```
4349

50+
```sh
51+
node on git:master ❯ ncu-ci rate commit
52+
--------------------------------------------------------------------------------
53+
[1/1] Running health
54+
--------------------------------------------------------------------------------
55+
✔ Done
56+
| UTC Time | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
57+
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
58+
| 2020-07-03 16:28 | 2 | 8 | 27 | 5 | 58 | 8.60% |
4459
```
45-
ncu-ci pr 12345
60+
61+
### `ncu-ci walk <type>`
62+
63+
`ncu-ci walk <type>` walks CI and displays failures, where `<type>` can be either `pr` for `node-test-pull-request` or `commit` for `node-test-commit`.
64+
65+
Example
66+
```sh
67+
node on git:master ❯ ncu-ci walk commit
68+
✔ Done--------------------------------------------------------------------------------
69+
[1/60] Running health
70+
--------------------------------------------------------------------------------
71+
| UTC Time | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
72+
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
73+
| 2020-07-03 16:50 | 1 | 8 | 27 | 5 | 59 | 8.51% |
74+
75+
--------------------------------------------------------------------------------
76+
[2/60] Running https://ci.nodejs.org/job/node-test-commit/39446/
77+
--------------------------------------------------------------------------------
78+
✔ Build data downloaded
79+
✔ Data downloaded
80+
----------------------------------- Summary ------------------------------------
81+
Result FAILURE
82+
URL https://ci.nodejs.org/job/node-test-commit/39446/
83+
Source https://api.github.com/repos/libuv/ci-tmp-libuv-node/git/refs/heads/jenkins-libuv-in-node-141
84+
Commit [b205f29c5b4d] Replace libuv version with refs/heads/v1.x from libuv/libuv
85+
Date 2020-07-03 15:44:21 +0000
86+
Author ci <[email protected]>
87+
-------------------------------- freebsd11-x64 ---------------------------------
88+
URL https://ci.nodejs.org/job/node-test-commit-freebsd/nodes=freebsd11-x64/34327/console
89+
Type JENKINS_FAILURE
90+
Built On test-digitalocean-freebsd11-x64-2
91+
Reason
92+
Build timed out (after 6 minutes). Marking the build as failed.
93+
--------------------------------- Other builds ---------------------------------
94+
Unstable https://ci.nodejs.org/job/node-test-commit-linux/35929/
95+
Unstable https://ci.nodejs.org/job/node-test-commit-arm-fanned/15212/
96+
--------------------------------------------------------------------------------
97+
[3/60] Running https://ci.nodejs.org/job/node-test-commit/39444/
98+
...etc
4699
```
47100

101+
Possible use cases:
48102

49-
Walk the CI for the latest 100 runs of `node-test-pull-request`,
50-
aggregate the failures, then write the results into a Markdown file,
51-
and cache the responses from Jenkins so that the next time the command
103+
1. Walk CI for the latest 100 runs of `node-test-pull-request`,
104+
aggregate failures, write the results into a Markdown file,
105+
and then cache the responses from Jenkins so that the next time the command
52106
is run, it picks up cached data written on disk for jobs whose results
53107
are known.
54108

55-
Note: results are cached in `${ncu_intallation_path}/.ncu/cache`, you
109+
Note: results are cached in `${ncu_intallation_path}/.ncu/cache`, so you
56110
may want to clean it up from time to time.
57111

58112
```
59113
ncu-ci walk pr --stats --cache --markdown results.md
60114
```
61115

62-
Walk the CI for the latest 100 runs of `node-test-pull-request`, write the
63-
failures into a JSON file
116+
2. Walk CI for the latest 100 runs of `node-test-pull-request`, and then write the
117+
failures into a JSON file named database.json.
64118

65119
```
66120
ncu-ci walk pr --json database.json
67121
```
68122

69-
Calculate the green rate of the CI for
70-
[CI Health History](https://github.com/nodejs/reliability#ci-health-history)
123+
### `ncu-ci pr <jobid>`
124+
125+
`ncu-ci pr <jobid>` returns information about the results of a `node-test-pull-request` job.
126+
127+
For example, if you would like to see the results of `node-test-pull-request` for https://github.com/nodejs/node/pull/34127, you would visit the PR and note that the `node-test-pull-request` job can be found at https://ci.nodejs.org/job/node-test-pull-request/32158, and therefore the `<jobid>` is `32158`, not `34127`.
128+
129+
Example:
130+
```sh
131+
node on git:master ❯ ncu-ci pr 32158
132+
--------------------------------------------------------------------------------
133+
[1/1] Running PR: 32158
134+
--------------------------------------------------------------------------------
135+
✔ Build data downloaded
136+
----------------------------------- Summary ------------------------------------
137+
Result UNSTABLE
138+
URL https://ci.nodejs.org/job/node-test-pull-request/32158/
139+
Source https://github.com/nodejs/node/pull/34127/
140+
Commit [faa808326cb0] lib: handle missing callbackMap in esm logic
141+
Date 2020-06-29 16:51:58 -0700
142+
Author Shelley Vohr <[email protected]>
143+
--------------------------------- Other builds ---------------------------------
144+
Unstable https://ci.nodejs.org/job/node-test-commit-arm-fanned/15155/
145+
```
146+
147+
### `ncu-ci commit <jobid>`
148+
149+
`ncu-ci commit <jobid>` returns information about the results of a specified `node-test-commit` job.
150+
151+
For example, if you would like to see the results of `node-test-commit` for https://github.com/nodejs/node/pull/34086, you would visit the PR and note that the `node-test-commit` job can be found at https://ci.nodejs.org/job/node-test-commit/39377, and therefore the `<jobid>` is `39377`, not `34086`.
152+
153+
Example:
154+
```sh
155+
node on git:master ❯ ncu-ci commit 39377
156+
--------------------------------------------------------------------------------
157+
[1/1] Running COMMIT: 39377
158+
--------------------------------------------------------------------------------
159+
✔ Build data downloaded
160+
----------------------------------- Summary ------------------------------------
161+
Result UNSTABLE
162+
URL https://ci.nodejs.org/job/node-test-commit/39377/
163+
Source https://github.com/undefined/undefined/pull/34086/
164+
Commit [725a1df7b849] quic: remove redundant cast
165+
Date 2020-06-27 22:14:09 +0800
166+
Author Jiawen Geng <[email protected]>
167+
--------------------------------- Other builds ---------------------------------
168+
Unstable https://ci.nodejs.org/job/node-test-commit-linux/35868/
169+
Unstable https://ci.nodejs.org/job/node-test-commit-arm-fanned/15148/
170+
```
71171

172+
### `ncu-ci url <url>`
173+
174+
`ncu-ci url <url>` takes a url corresponding to a PR on `nodejs/node`, detects the CI type (either `node-test-commit` or `node-test-pull-request`) and corresponding job id for the latest run, and returns a summary of results about the job run.
175+
176+
Example:
177+
```sh
178+
node on git:master ❯ ncu-ci url https://github.com/nodejs/node/pull/34127
179+
--------------------------------------------------------------------------------
180+
[1/1] Running PR: 32158
181+
--------------------------------------------------------------------------------
182+
✔ Build data downloaded
183+
----------------------------------- Summary ------------------------------------
184+
Result UNSTABLE
185+
URL https://ci.nodejs.org/job/node-test-pull-request/32158/
186+
Source https://github.com/nodejs/node/pull/34127/
187+
Commit [faa808326cb0] lib: handle missing callbackMap in esm logic
188+
Date 2020-06-29 16:51:58 -0700
189+
Author Shelley Vohr <[email protected]>
190+
--------------------------------- Other builds ---------------------------------
191+
Unstable https://ci.nodejs.org/job/node-test-commit-arm-fanned/15155/
72192
```
73-
ncu-ci rate pr
193+
194+
### `ncu-ci benchmark <jobid>`
195+
196+
`ncu-ci benchmark <jobid>` displays the results of a specified `benchmark-node-micro-benchmarks` CI job.
197+
198+
Example:
199+
```sh
200+
node on git:master ❯ ncu-ci benchmark 636
201+
--------------------------------------------------------------------------------
202+
[1/1] Running BENCHMARK: 636
203+
--------------------------------------------------------------------------------
204+
✔ Data downloaded
205+
confidence improvement accuracy (*) (**) (***)
206+
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=100 requests=100 NA NaN % NA NA NA
207+
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=10 requests=100 NA NaN % NA NA NA
208+
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=10 requests=1000 NA NaN % NA NA NA
209+
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=1 requests=100 NA NaN % NA NA NA
210+
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=1 requests=1000 NA NaN % NA NA NA
211+
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=200 requests=100 NA NaN % NA NA NA
212+
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=20 requests=100 NA NaN % NA NA NA
213+
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=20 requests=1000 NA NaN % NA NA NA
214+
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=40 requests=100 NA NaN % NA NA NA
215+
216+
Be aware that when doing many comparisons the risk of a false-positive
217+
result increases. In this case there are 9 comparisons, you can thus
218+
expect the following amount of false-positive results:
219+
0.45 false positives, when considering a 5% risk acceptance (*, **, ***),
220+
0.09 false positives, when considering a 1% risk acceptance (**, ***),
221+
0.01 false positives, when considering a 0.1% risk acceptance (***)
222+
++ mv output030720-170033.csv /w/bnch-comp
223+
Collecting metadata...
224+
Metadata collection done.
225+
Notifying upstream projects of job completion
226+
Finished: SUCCESS
74227
```
75228
76229
## Caveats

0 commit comments

Comments
 (0)