Skip to content

Commit ad47e7d

Browse files
wkingwking
committed
Makefile: Change from prove to node-tap
For better reporting, including skips. Prove (at least as of TAP::Harness v3.35_01 and Perl v5.22.3) seems to report skips only when there were also failures. For example, here's a skip-only test: $ validation/mounts.t TAP version 13 1..1 ok 1 # SKIP TODO: mounts generation options have not been implemented $ prove validation/mounts.t validation/mounts.t .. ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU) Result: PASS node-tap (as of version 11.0.0) reports that skip: $ tap validation/mounts.t validation/mounts.t ................................... 0/1 Skipped: 1 TODO: mounts generation options have not been implemented total ................................................. 0/1 0 passing (27.297ms) 1 pending And here's a skip with a failure test: $ ./test-skip TAP version 13 1..2 not ok 1 - failing ok 2 # SKIP: skipping Prove warns about the skip now: $ prove test-skip test-skip .. Failed 1/2 subtests (less 1 skipped subtest: 0 okay) Test Summary Report ------------------- test-skip (Wstat: 0 Tests: 2 Failed: 1) Failed test: 1 Files=1, Tests=2, 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU) Result: FAIL But node-tap has a nicer warning: $ tap ./test-skip ./test-skip ........................................... 0/2 not ok failing Skipped: 1 : skipping total ................................................. 0/2 0 passing (39.088ms) 1 pending 1 failing Similarly, node-tap does a better job handling YAML blocks [1]: $ ./test-yaml TAP version 13 1..4 ok 1 - success diagnostic # success not ok 2 - failure diagnostic # failure ok 3 - success YAML --- message: success ... not ok 4 - failure YAML --- message: failure ... Prove either shows no diagnostics: $ prove test-yaml test-yaml .. Failed 2/4 subtests Test Summary Report ------------------- test-yaml (Wstat: 0 Tests: 4 Failed: 2) Failed tests: 2, 4 Files=1, Tests=4, 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU) Result: FAIL or it shows all the diagnostics (even for successful tests): $ prove --comments test-yaml test-yaml .. 1/? # success # failure test-yaml .. Failed 2/4 subtests Test Summary Report ------------------- test-yaml (Wstat: 0 Tests: 4 Failed: 2) Failed tests: 2, 4 Files=1, Tests=4, 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU) Result: FAIL I can't find a way to get Prove to show the YAML blocks. node-tap, on the other hand, does the right thing with YAML blocks: $ tap ./test-yaml ./test-yaml ........................................... 2/4 not ok failure diagnostic not ok failure YAML message: failure total ................................................. 2/4 2 passing (42.409ms) 2 failing We don't use YAML blocks at the moment, but I'm going to transition to them later. [1]: http://testanything.org/tap-version-13-specification.html#yaml-blocks Signed-off-by: W. Trevor King <[email protected]>
1 parent 0a919c0 commit ad47e7d

File tree

2 files changed

+108
-60
lines changed

2 files changed

+108
-60
lines changed

‎Makefile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
PREFIX ?= $(DESTDIR)/usr
22
BINDIR ?= $(DESTDIR)/usr/bin
3+
TAP ?= tap
34

45
BUILDTAGS=
56
RUNTIME ?= runc
@@ -39,7 +40,7 @@ clean:
3940
rm -f oci-runtime-tool runtimetest *.1 $(VALIDATION_TESTS)
4041

4142
localvalidation:
42-
RUNTIME=$(RUNTIME) prove $(VALIDATION_TESTS)
43+
RUNTIME=$(RUNTIME) $(TAP) $(VALIDATION_TESTS)
4344

4445
.PHONY: validation-executables
4546
validation-executables: $(VALIDATION_TESTS)

‎README.md‎

Lines changed: 106 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -30,72 +30,84 @@ INFO[0000] Bundle validation succeeded.
3030

3131
## Testing OCI runtimes
3232

33-
The runtime validation suite uses [`prove`][prove], which is packaged for most distributions (for example, it is in [Debian's `perl` package][debian-perl] and [Gentoo's `dev-lang/perl` package][gentoo-perl]).
34-
If you cannot install `prove`, you can probably run the test suite with another [TAP consumer][tap-consumers], although you'll have to edit the [`Makefile`](Makefile) to replace `prove`.
33+
The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]).
34+
If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it:
35+
36+
```console
37+
$ npm install tap
38+
```
3539

3640
```console
3741
$ make runtimetest validation-executables
38-
$ sudo make RUNTIME=runc localvalidation
39-
RUNTIME=runc prove validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
40-
validation/linux_rootfs_propagation_shared.t ...... Failed 1/19 subtests
41-
validation/create.t ............................... ok
42-
validation/default.t .............................. ok
43-
validation/linux_readonly_paths.t ................. ok
44-
validation/linux_masked_paths.t ................... Failed 1/19 subtests
45-
validation/mounts.t ............................... ok
46-
validation/process.t .............................. ok
47-
validation/root_readonly_false.t .................. ok
48-
validation/linux_sysctl.t ......................... ok
49-
validation/linux_devices.t ........................ ok
50-
validation/linux_gid_mappings.t ................... Failed 1/19 subtests
51-
validation/process_oom_score_adj.t ................ ok
52-
validation/process_capabilities.t ................. ok
53-
validation/process_rlimits.t ...................... ok
54-
validation/root_readonly_true.t ................... ok
55-
validation/linux_rootfs_propagation_unbindable.t .. failed to create the container
42+
RUNTIME=runc tap validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
43+
validation/linux_rootfs_propagation_shared.t ........ 18/19
44+
not ok rootfs propagation
45+
46+
validation/create.t ................................... 4/4
47+
validation/default.t ................................ 19/19
48+
validation/linux_readonly_paths.t ................... 19/19
49+
validation/linux_masked_paths.t ..................... 18/19
50+
not ok masked paths
51+
52+
validation/mounts.t ................................... 0/1
53+
Skipped: 1
54+
TODO: mounts generation options have not been implemented
55+
56+
validation/process.t ................................ 19/19
57+
validation/root_readonly_false.t .................... 19/19
58+
validation/linux_sysctl.t ........................... 19/19
59+
validation/linux_devices.t .......................... 19/19
60+
validation/linux_gid_mappings.t ..................... 18/19
61+
not ok gid mappings
62+
63+
validation/process_oom_score_adj.t .................. 19/19
64+
validation/process_capabilities.t ................... 19/19
65+
validation/process_rlimits.t ........................ 19/19
66+
validation/root_readonly_true.t ...................failed to create the container
5667
rootfsPropagation=unbindable is not supported
5768
exit status 1
58-
validation/linux_rootfs_propagation_unbindable.t .. Dubious, test returned 1 (wstat 256, 0x100)
59-
No subtests run
60-
validation/hostname.t ............................. ok
61-
validation/linux_uid_mappings.t ................... failed to create the container
69+
validation/root_readonly_true.t ..................... 19/19
70+
validation/linux_rootfs_propagation_unbindable.t ...... 0/1
71+
not ok validation/linux_rootfs_propagation_unbindable.t
72+
timeout: 30000
73+
file: validation/linux_rootfs_propagation_unbindable.t
74+
command: validation/linux_rootfs_propagation_unbindable.t
75+
args: []
76+
stdio:
77+
- 0
78+
- pipe
79+
- 2
80+
cwd: /…/go/src/github.com/opencontainers/runtime-tools
81+
exitCode: 1
82+
83+
validation/hostname.t ...................failed to create the container
6284
User namespace mappings specified, but USER namespace isn't enabled in the config
6385
exit status 1
64-
validation/linux_uid_mappings.t ................... Dubious, test returned 1 (wstat 256, 0x100)
65-
No subtests run
66-
67-
Test Summary Report
68-
-------------------
69-
validation/linux_rootfs_propagation_shared.t (Wstat: 0 Tests: 19 Failed: 1)
70-
Failed test: 16
71-
validation/linux_masked_paths.t (Wstat: 0 Tests: 19 Failed: 1)
72-
Failed test: 13
73-
validation/linux_gid_mappings.t (Wstat: 0 Tests: 19 Failed: 1)
74-
Failed test: 19
75-
validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0)
76-
Non-zero exit status: 1
77-
Parse errors: No plan found in TAP output
78-
validation/linux_uid_mappings.t (Wstat: 256 Tests: 0 Failed: 0)
79-
Non-zero exit status: 1
80-
Parse errors: No plan found in TAP output
81-
Files=18, Tests=271, 31 wallclock secs ( 0.06 usr 0.01 sys + 0.52 cusr 0.21 csys = 0.80 CPU)
82-
Result: FAIL
83-
make: *** [Makefile:42: localvalidation] Error 1
86+
validation/hostname.t ............................... 19/19
87+
validation/linux_uid_mappings.t ....................... 0/1
88+
not ok validation/linux_uid_mappings.t
89+
timeout: 30000
90+
file: validation/linux_uid_mappings.t
91+
command: validation/linux_uid_mappings.t
92+
args: []
93+
stdio:
94+
- 0
95+
- pipe
96+
- 2
97+
cwd: /…/go/src/github.com/opencontainers/runtime-tools
98+
exitCode: 1
99+
100+
total ............................................. 267/273
101+
102+
103+
267 passing (31s)
104+
1 pending
105+
5 failing
106+
107+
make: *** [Makefile:43: localvalidation] Error 1
84108
```
85109

86-
If you are confident that the `validation/*.t` are current, you can run `prove` (or your preferred TAP consumer) directly to avoid unnecessary rebuilds:
87-
88-
```console
89-
$ sudo RUNTIME=runc prove -Q validation/*.t
90-
91-
Test Summary Report
92-
-------------------
93-
…
94-
Files=18, Tests=271, 31 wallclock secs ( 0.07 usr 0.01 sys + 0.51 cusr 0.22 csys = 0.81 CPU)
95-
Result: FAIL
96-
```
97-
98-
And you can run an individual test executable directly:
110+
You can also run an individual test executable directly:
99111

100112
```console
101113
$ RUNTIME=runc validation/default.t
@@ -122,10 +134,45 @@ ok 19 - gid mappings
122134
1..19
123135
```
124136

137+
If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers].
138+
For example, with [`prove`][prove]:
139+
140+
```console
141+
$ sudo make TAP='prove -Q -j9' RUNTIME=runc localvalidation
142+
RUNTIME=runc prove -Q -j9 validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
143+
failed to create the container
144+
rootfsPropagation=unbindable is not supported
145+
exit status 1
146+
failed to create the container
147+
User namespace mappings specified, but USER namespace isn't enabled in the config
148+
exit status 1
149+
150+
Test Summary Report
151+
-------------------
152+
validation/linux_rootfs_propagation_shared.t (Wstat: 0 Tests: 19 Failed: 1)
153+
Failed test: 16
154+
validation/linux_masked_paths.t (Wstat: 0 Tests: 19 Failed: 1)
155+
Failed test: 13
156+
validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0)
157+
Non-zero exit status: 1
158+
Parse errors: No plan found in TAP output
159+
validation/linux_uid_mappings.t (Wstat: 256 Tests: 0 Failed: 0)
160+
Non-zero exit status: 1
161+
Parse errors: No plan found in TAP output
162+
validation/linux_gid_mappings.t (Wstat: 0 Tests: 19 Failed: 1)
163+
Failed test: 19
164+
Files=18, Tests=271, 6 wallclock secs ( 0.06 usr 0.01 sys + 0.59 cusr 0.24 csys = 0.90 CPU)
165+
Result: FAIL
166+
make: *** [Makefile:43: localvalidation] Error 1
167+
```
168+
125169
[bundle]: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md
126170
[config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md
127-
[debian-perl]: https://packages.debian.org/stretch/perl
128-
[gentoo-perl]: https://packages.gentoo.org/packages/dev-lang/perl
171+
[debian-node-tap]: https://packages.debian.org/stretch/node-tap
172+
[debian-nodejs]: https://packages.debian.org/stretch/nodejs
173+
[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs
174+
[node-tap]: http://www.node-tap.org/
175+
[npm]: https://www.npmjs.com/
129176
[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove
130177
[runC]: https://github.com/opencontainers/runc
131178
[runtime-spec]: https://github.com/opencontainers/runtime-spec

0 commit comments

Comments
 (0)