Skip to content

Commit 294ca2a

Browse files
albanDongsu Park
alban
authored and
Dongsu Park
committed
docs: move compliance testing to separate doc
Signed-off-by: Alban Crequy <[email protected]>
1 parent 7aaee9c commit 294ca2a

File tree

2 files changed

+159
-89
lines changed

2 files changed

+159
-89
lines changed

‎README.md‎

Lines changed: 1 addition & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -30,99 +30,11 @@ INFO[0000] Bundle validation succeeded.
3030

3131
## Testing OCI runtimes
3232

33-
The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]).
34-
If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it:
35-
36-
```console
37-
$ npm install tap
38-
```
39-
40-
Build the validation executables:
41-
42-
```console
43-
$ make runtimetest validation-executables
44-
```
45-
46-
Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](docs/command-line-interface.md).
47-
If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API.
48-
For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](docs/command-line-interface.md)).
49-
50-
```
51-
$ sudo make RUNTIME=runc localvalidation
52-
RUNTIME=runc tap validation/pidfile.t validation/linux_cgroups_hugetlb.t validation/linux_cgroups_memory.t validation/linux_rootfs_propagation_shared.t validation/kill.t validation/create.t validation/poststart.t validation/linux_cgroups_network.t validation/poststop_fail.t validation/linux_readonly_paths.t validation/prestart_fail.t validation/hooks_stdin.t validation/default.t validation/linux_masked_paths.t validation/poststop.t validation/misc_props.t validation/prestart.t validation/poststart_fail.t validation/mounts.t validation/linux_cgroups_relative_pids.t validation/process_user.t validation/process.t validation/hooks.t validation/process_capabilities_fail.t validation/process_rlimits_fail.t validation/linux_cgroups_relative_cpus.t validation/process_rlimits.t validation/linux_cgroups_relative_blkio.t validation/linux_sysctl.t validation/linux_seccomp.t validation/linux_devices.t validation/start.t validation/linux_cgroups_pids.t validation/process_capabilities.t validation/process_oom_score_adj.t validation/linux_cgroups_relative_hugetlb.t validation/linux_cgroups_cpus.t validation/linux_cgroups_relative_memory.t validation/state.t validation/root_readonly_true.t validation/linux_cgroups_blkio.t validation/linux_rootfs_propagation_unbindable.t validation/delete.t validation/linux_cgroups_relative_network.t validation/hostname.t validation/killsig.t validation/linux_uid_mappings.t
53-
validation/pidfile.t .failed to create the container
54-
container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:367: setting cgroup config for procHooks process caused \\\"failed to write 56892210544640 to hugetlb.1GB.limit_in_bytes: open /sys/fs/cgroup/hugetlb/cgrouptest/hugetlb.1GB.limit_in_bytes: permission denied\\\"\""
55-
exit status 1
56-
validation/pidfile.t .................................. 1/1 315ms
57-
validation/linux_cgroups_hugetlb.t .................... 0/1
58-
not ok validation/linux_cgroups_hugetlb.t
59-
timeout: 30000
60-
file: validation/linux_cgroups_hugetlb.t
61-
command: validation/linux_cgroups_hugetlb.t
62-
args: []
63-
stdio:
64-
- 0
65-
- pipe
66-
- 2
67-
cwd: /…/go/src/github.com/opencontainers/runtime-tools
68-
exitCode: 1
69-
70-
validation/linux_cgroups_memory.t ..................... 9/9
71-
validation/linux_rootfs_propagation_shared.t ...... 252/282
72-
not ok shared root propagation exposes "/target348456609/mount892511628/example376408222"
73-
74-
Skipped: 29
75-
/dev/null (default device) has unconfigured permissions
76-
…
77-
total ........................................... 4381/4962
78-
79-
80-
4381 passing (1m)
81-
567 pending
82-
14 failing
83-
84-
make: *** [Makefile:44: localvalidation] Error 1
85-
```
86-
87-
You can also run an individual test executable directly:
88-
89-
```console
90-
$ RUNTIME=runc validation/default.t
91-
TAP version 13
92-
ok 1 - has expected hostname
93-
---
94-
{
95-
"actual": "mrsdalloway",
96-
"expected": "mrsdalloway"
97-
}
98-
...
99-
…
100-
ok 287 # SKIP linux.gidMappings not set
101-
1..287
102-
```
103-
104-
If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers].
105-
For example, with [`prove`][prove]:
106-
107-
```console
108-
$ sudo make TAP='prove -Q -j9' RUNTIME=runc VALIDATION_TESTS=validation/pidfile.t localvalidation
109-
RUNTIME=runc prove -Q -j9 validation/pidfile.t
110-
All tests successful.
111-
Files=1, Tests=1, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.03 cusr 0.03 csys = 0.08 CPU)
112-
Result: PASS
113-
```
33+
Use the [runtime validation suite](doc/runtime-compliance-testing.md).
11434

11535
[bundle]: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md
11636
[config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md
117-
[debian-node-tap]: https://packages.debian.org/stretch/node-tap
118-
[debian-nodejs]: https://packages.debian.org/stretch/nodejs
119-
[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs
120-
[node-tap]: http://www.node-tap.org/
121-
[npm]: https://www.npmjs.com/
122-
[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove
12337
[runC]: https://github.com/opencontainers/runc
12438
[runtime-spec]: https://github.com/opencontainers/runtime-spec
125-
[tap-consumers]: https://testanything.org/consumers.html
126-
12739
[generate.1]: man/oci-runtime-tool-generate.1.md
12840
[validate.1]: man/oci-runtime-tool-validate.1.md

‎docs/runtime-compliance-testing.md‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,162 @@ In order to be tested for [compliance][], runtimes MUST support at least one of
66

77
* Version 1.0.1 of the [OCI Runtime Command Line Interface](command-line-interface.md).
88

9+
## Running the runtime validation suite from source
10+
11+
The runtime validation suite uses [node-tap][], which is packaged for some distributions (for example, it is in [Debian's `node-tap` package][debian-node-tap]).
12+
If your distribution does not package node-tap, you can install [npm][] (for example, from [Gentoo's `nodejs` package][gentoo-nodejs]) and use it:
13+
14+
```console
15+
$ npm install tap
16+
```
17+
18+
Build the validation executables:
19+
20+
```console
21+
$ make runtimetest validation-executables
22+
```
23+
24+
Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](doc/command-line-interface.md).
25+
If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API.
26+
For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](doc/command-line-interface.md)).
27+
28+
```
29+
$ sudo make RUNTIME=runc localvalidation
30+
RUNTIME=runc tap validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
31+
validation/linux_rootfs_propagation_shared.t ........ 18/19
32+
not ok rootfs propagation
33+
error: 'rootfs should be shared, but not'
34+
35+
validation/create.t ................................... 4/4
36+
validation/default.t ................................ 19/19
37+
validation/linux_readonly_paths.t ................... 19/19
38+
validation/linux_masked_paths.t ..................... 18/19
39+
not ok masked paths
40+
error: /masktest should not be readable
41+
42+
validation/mounts.t ................................... 0/1
43+
Skipped: 1
44+
TODO: mounts generation options have not been implemented
45+
46+
validation/process.t ................................ 19/19
47+
validation/root_readonly_false.t .................... 19/19
48+
validation/linux_sysctl.t ........................... 19/19
49+
validation/linux_devices.t .......................... 19/19
50+
validation/linux_gid_mappings.t ..................... 18/19
51+
not ok gid mappings
52+
53+
validation/process_oom_score_adj.t .................. 19/19
54+
validation/process_capabilities.t ................... 19/19
55+
validation/process_rlimits.t ........................ 19/19
56+
validation/root_readonly_true.t ...................failed to create the container
57+
rootfsPropagation=unbindable is not supported
58+
exit status 1
59+
validation/root_readonly_true.t ..................... 19/19
60+
validation/linux_rootfs_propagation_unbindable.t ...... 0/1
61+
not ok validation/linux_rootfs_propagation_unbindable.t
62+
timeout: 30000
63+
file: validation/linux_rootfs_propagation_unbindable.t
64+
command: validation/linux_rootfs_propagation_unbindable.t
65+
args: []
66+
stdio:
67+
- 0
68+
- pipe
69+
- 2
70+
cwd: /…/go/src/github.com/opencontainers/runtime-tools
71+
exitCode: 1
72+
73+
validation/hostname.t ...................failed to create the container
74+
User namespace mappings specified, but USER namespace isn't enabled in the config
75+
exit status 1
76+
validation/hostname.t ............................... 19/19
77+
validation/linux_uid_mappings.t ....................... 0/1
78+
not ok validation/linux_uid_mappings.t
79+
timeout: 30000
80+
file: validation/linux_uid_mappings.t
81+
command: validation/linux_uid_mappings.t
82+
args: []
83+
stdio:
84+
- 0
85+
- pipe
86+
- 2
87+
cwd: /…/go/src/github.com/opencontainers/runtime-tools
88+
exitCode: 1
89+
90+
total ............................................. 267/273
91+
92+
93+
267 passing (31s)
94+
1 pending
95+
5 failing
96+
97+
make: *** [Makefile:43: localvalidation] Error 1
98+
```
99+
100+
You can also run an individual test executable directly:
101+
102+
```console
103+
$ RUNTIME=runc validation/default.t
104+
TAP version 13
105+
ok 1 - root filesystem
106+
ok 2 - hostname
107+
ok 3 - process
108+
ok 4 - mounts
109+
ok 5 - user
110+
ok 6 - rlimits
111+
ok 7 - capabilities
112+
ok 8 - default symlinks
113+
ok 9 - default file system
114+
ok 10 - default devices
115+
ok 11 - linux devices
116+
ok 12 - linux process
117+
ok 13 - masked paths
118+
ok 14 - oom score adj
119+
ok 15 - read only paths
120+
ok 16 - rootfs propagation
121+
ok 17 - sysctls
122+
ok 18 - uid mappings
123+
ok 19 - gid mappings
124+
1..19
125+
```
126+
127+
If you cannot install node-tap, you can probably run the test suite with another [TAP consumer][tap-consumers].
128+
For example, with [`prove`][prove]:
129+
130+
```console
131+
$ sudo make TAP='prove -Q -j9' RUNTIME=runc localvalidation
132+
RUNTIME=runc prove -Q -j9 validation/linux_rootfs_propagation_shared.t validation/create.t validation/default.t validation/linux_readonly_paths.t validation/linux_masked_paths.t validation/mounts.t validation/process.t validation/root_readonly_false.t validation/linux_sysctl.t validation/linux_devices.t validation/linux_gid_mappings.t validation/process_oom_score_adj.t validation/process_capabilities.t validation/process_rlimits.t validation/root_readonly_true.t validation/linux_rootfs_propagation_unbindable.t validation/hostname.t validation/linux_uid_mappings.t
133+
failed to create the container
134+
rootfsPropagation=unbindable is not supported
135+
exit status 1
136+
failed to create the container
137+
User namespace mappings specified, but USER namespace isn't enabled in the config
138+
exit status 1
139+
140+
Test Summary Report
141+
-------------------
142+
validation/linux_rootfs_propagation_shared.t (Wstat: 0 Tests: 19 Failed: 1)
143+
Failed test: 16
144+
validation/linux_masked_paths.t (Wstat: 0 Tests: 19 Failed: 1)
145+
Failed test: 13
146+
validation/linux_rootfs_propagation_unbindable.t (Wstat: 256 Tests: 0 Failed: 0)
147+
Non-zero exit status: 1
148+
Parse errors: No plan found in TAP output
149+
validation/linux_uid_mappings.t (Wstat: 256 Tests: 0 Failed: 0)
150+
Non-zero exit status: 1
151+
Parse errors: No plan found in TAP output
152+
validation/linux_gid_mappings.t (Wstat: 0 Tests: 19 Failed: 1)
153+
Failed test: 19
154+
Files=18, Tests=271, 6 wallclock secs ( 0.06 usr 0.01 sys + 0.59 cusr 0.24 csys = 0.90 CPU)
155+
Result: FAIL
156+
make: *** [Makefile:43: localvalidation] Error 1
157+
```
158+
159+
9160
[compliance]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/spec.md
161+
[debian-node-tap]: https://packages.debian.org/stretch/node-tap
162+
[debian-nodejs]: https://packages.debian.org/stretch/nodejs
163+
[gentoo-nodejs]: https://packages.gentoo.org/packages/net-libs/nodejs
164+
[node-tap]: http://www.node-tap.org/
165+
[npm]: https://www.npmjs.com/
166+
[prove]: http://search.cpan.org/~leont/Test-Harness-3.39/bin/prove
167+
[tap-consumers]: https://testanything.org/consumers.html

0 commit comments

Comments
 (0)