Skip to content

Commit 346b492

Browse files
apostasieapostasie
committed
Test tooling fixes
As we make progress rewriting tests, the new tooling needs to adapt. In a shell, this is: - introducing (more) `Requirements`, with a better API - update documentation - fix some t.Helper calls - fix broken stdin implementation - do cleanup custom namespaces properly - change hashing function - disable "private" implying custom data root which is more trouble than is worth - minor cleanups Signed-off-by: apostasie <[email protected]>
1 parent ca69b10 commit 346b492

File tree

9 files changed

+411
-200
lines changed

9 files changed

+411
-200
lines changed

‎docs/testing/tools.md

Lines changed: 48 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
The integration test suite in nerdctl is meant to apply to both nerdctl and docker,
66
and further support additional test properties to target specific contexts (ipv6, kube).
77

8-
Basic _usage_ is covered in the [testing docs](testing.md).
8+
Basic _usage_ is covered in the [testing docs](README.md).
99

1010
This here covers how to write tests, leveraging nerdctl `pkg/testutil/test`
1111
which has been specifically developed to take care of repetitive tasks,
@@ -166,9 +166,6 @@ Note that `Data` additionally exposes the following functions:
166166
Secondly, `Data` allows defining and manipulating "configuration" data.
167167

168168
In the case of nerdctl here, the following configuration options are defined:
169-
- `WithConfig(Docker, NotCompatible)` to flag a test as not compatible
170-
- `WithConfig(Mode, Private)` will entirely isolate the test using a different
171-
namespace, data root, nerdctl config, etc
172169
- `WithConfig(NerdctlToml, "foo")` which allows specifying a custom config
173170
- `WithConfig(DataRoot, "foo")` allowing to point to a custom data-root
174171
- `WithConfig(HostsDir, "foo")` to point to a specific hosts directory
@@ -358,7 +355,50 @@ All tests (and subtests) are assumed to be parallelizable.
358355
You can force a specific `test.Case` to not be run in parallel though,
359356
by setting its `NoParallel` property to `true`.
360357

361-
Note that if you want better isolation, it is usually better to use
362-
`WithConfig(nerdtest.Mode, nerdtest.Private)` instead.
363-
This will keep the test parallel (for nerdctl), but isolate it in a different context.
364-
For Docker (which does not support namespaces), it is equivalent to passing `NoParallel: true`.
358+
Note that if you want better isolation, it is usually better to use the requirement
359+
`nerdtest.Private` instead of `NoParallel` (see below).
360+
361+
## Requirements
362+
363+
`test.Case` has a `Require` property that allow enforcing specific, per-test requirements.
364+
365+
Here are a few:
366+
```go
367+
test.Windows // a test runs only on Windows (or Not(Windows))
368+
test.Linux // a test runs only on Linux
369+
test.Darwin // a test runs only on Darwin
370+
test.OS(name string) // a test runs only on the OS `name`
371+
test.Binary(name string) // a test requires the bin `name` to be in the PATH
372+
test.Not(req Requirement) // a test runs only if the opposite of the requirement `req` is fulfilled
373+
test.Require(req ...Requirement) // a test runs only if all requirements are fulfilled
374+
375+
nerdtest.Docker // a test only run on Docker - normally used with test.Not(nerdtest.Docker)
376+
nerdtest.Soci // a test requires the soci snapshotter
377+
nerdtest.Rootless // a test requires Rootless (or Not(Rootless), indicating it requires Rootful)
378+
nerdtest.Build // a test requires buildkit
379+
nerdtest.CGroup // a test requires cgroup
380+
nerdtest.OnlyIPv6 // a test is meant to run solely in the ipv6 environment
381+
nerdtest.NerdctlNeedsFixing // indicates that a test cannot be run on nerdctl yet as a fix is required
382+
nerdtest.Private // see below
383+
```
384+
385+
### About `nerdtest.Private`
386+
387+
While all requirements above are self-descriptive or obvious, and are going to skip
388+
tests for environments that do not match the requirements, `nerdtest.Private` is a
389+
special case.
390+
391+
What it does when required is: create a private namespace, data-root, hosts-dir, nerdctl.toml and
392+
DOCKER_CONFIG that is private to the test.
393+
394+
Note that subtests are going to inherit that environment as well.
395+
396+
If the target is Docker - which does not support namespaces for eg - asking for `private`
397+
will merely disable parallelization.
398+
399+
The purpose of private is to provide a truly clean-room environment for tests
400+
that are guaranteed to have side effects on others, or that do require an exclusive, pristine
401+
environment.
402+
403+
Using private is generally preferable to disabling parallelization, as doing the latter
404+
would slow down the run and won't have the same guarantees about the environment.

‎pkg/testutil/nerdtest/helpers.go

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
/*
2+
Copyright The containerd Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package nerdtest
18+
19+
import (
20+
"encoding/json"
21+
"testing"
22+
"time"
23+
24+
"gotest.tools/v3/assert"
25+
26+
"github.com/containerd/nerdctl/v2/pkg/inspecttypes/dockercompat"
27+
"github.com/containerd/nerdctl/v2/pkg/inspecttypes/native"
28+
"github.com/containerd/nerdctl/v2/pkg/testutil/test"
29+
)
30+
31+
// InspectContainer is a helper that can be used inside custom commands or Setup
32+
func InspectContainer(helpers test.Helpers, name string) dockercompat.Container {
33+
var dc []dockercompat.Container
34+
cmd := helpers.Command("container", "inspect", name)
35+
cmd.Run(&test.Expected{
36+
ExitCode: 0,
37+
Output: func(stdout string, info string, t *testing.T) {
38+
err := json.Unmarshal([]byte(stdout), &dc)
39+
assert.NilError(t, err, "Unable to unmarshal output\n"+info)
40+
assert.Equal(t, 1, len(dc), "Unexpectedly got multiple results\n"+info)
41+
},
42+
})
43+
return dc[0]
44+
}
45+
46+
func InspectVolume(helpers test.Helpers, name string, args ...string) native.Volume {
47+
var dc []native.Volume
48+
cmdArgs := append([]string{"volume", "inspect"}, args...)
49+
cmdArgs = append(cmdArgs, name)
50+
51+
cmd := helpers.Command(cmdArgs...)
52+
cmd.Run(&test.Expected{
53+
ExitCode: 0,
54+
Output: func(stdout string, info string, t *testing.T) {
55+
err := json.Unmarshal([]byte(stdout), &dc)
56+
assert.NilError(t, err, "Unable to unmarshal output\n"+info)
57+
assert.Equal(t, 1, len(dc), "Unexpectedly got multiple results\n"+info)
58+
},
59+
})
60+
return dc[0]
61+
}
62+
63+
func InspectNetwork(helpers test.Helpers, name string, args ...string) dockercompat.Network {
64+
var dc []dockercompat.Network
65+
cmdArgs := append([]string{"network", "inspect"}, args...)
66+
cmdArgs = append(cmdArgs, name)
67+
68+
cmd := helpers.Command(cmdArgs...)
69+
cmd.Run(&test.Expected{
70+
ExitCode: 0,
71+
Output: func(stdout string, info string, t *testing.T) {
72+
err := json.Unmarshal([]byte(stdout), &dc)
73+
assert.NilError(t, err, "Unable to unmarshal output\n"+info)
74+
assert.Equal(t, 1, len(dc), "Unexpectedly got multiple results\n"+info)
75+
},
76+
})
77+
return dc[0]
78+
}
79+
80+
func InspectImage(helpers test.Helpers, name string) dockercompat.Image {
81+
var dc []dockercompat.Image
82+
cmd := helpers.Command("image", "inspect", name)
83+
cmd.Run(&test.Expected{
84+
ExitCode: 0,
85+
Output: func(stdout string, info string, t *testing.T) {
86+
err := json.Unmarshal([]byte(stdout), &dc)
87+
assert.NilError(t, err, "Unable to unmarshal output\n"+info)
88+
assert.Equal(t, 1, len(dc), "Unexpectedly got multiple results\n"+info)
89+
},
90+
})
91+
return dc[0]
92+
}
93+
94+
func EnsureContainerStarted(helpers test.Helpers, con string) {
95+
const (
96+
maxRetry = 5
97+
sleep = time.Second
98+
)
99+
for i := 0; i < maxRetry; i++ {
100+
count := i
101+
cmd := helpers.Command("container", "inspect", con)
102+
cmd.Run(&test.Expected{
103+
ExitCode: 0,
104+
Output: func(stdout string, info string, t *testing.T) {
105+
var dc []dockercompat.Container
106+
err := json.Unmarshal([]byte(stdout), &dc)
107+
assert.NilError(t, err, "Unable to unmarshal output\n"+info)
108+
assert.Equal(t, 1, len(dc), "Unexpectedly got multiple results\n"+info)
109+
if dc[0].State.Running {
110+
return
111+
}
112+
if count == maxRetry-1 {
113+
t.Fatalf("conainer %s not running", con)
114+
}
115+
time.Sleep(sleep)
116+
},
117+
})
118+
}
119+
}

‎pkg/testutil/nerdtest/requirements.go

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
/*
2+
Copyright The containerd Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package nerdtest
18+
19+
import (
20+
"encoding/json"
21+
"fmt"
22+
"testing"
23+
24+
"gotest.tools/v3/assert"
25+
26+
"github.com/containerd/nerdctl/v2/pkg/buildkitutil"
27+
"github.com/containerd/nerdctl/v2/pkg/inspecttypes/dockercompat"
28+
"github.com/containerd/nerdctl/v2/pkg/rootlessutil"
29+
"github.com/containerd/nerdctl/v2/pkg/testutil"
30+
"github.com/containerd/nerdctl/v2/pkg/testutil/test"
31+
)
32+
33+
var ipv6 test.ConfigKey = "IPv6Test"
34+
var only test.ConfigValue = "Only"
35+
var mode test.ConfigKey = "Mode"
36+
var modePrivate test.ConfigValue = "Private"
37+
38+
var OnlyIPv6 = test.MakeRequirement(func(data test.Data, t *testing.T) (ret bool, mess string) {
39+
ret = testutil.GetEnableIPv6()
40+
if !ret {
41+
mess = "runner skips IPv6 compatible tests in the non-IPv6 environment"
42+
}
43+
data.WithConfig(ipv6, only)
44+
return ret, mess
45+
})
46+
47+
var Private = test.MakeRequirement(func(data test.Data, t *testing.T) (ret bool, mess string) {
48+
data.WithConfig(mode, modePrivate)
49+
return true, "private mode"
50+
})
51+
52+
var Soci = test.MakeRequirement(func(data test.Data, t *testing.T) (ret bool, mess string) {
53+
ret = false
54+
mess = "soci is not enabled"
55+
(&test.GenericCommand{}).
56+
WithT(t).
57+
WithBinary(testutil.GetTarget()).
58+
WithArgs("info", "--format", "{{ json . }}").
59+
Run(&test.Expected{
60+
Output: func(stdout string, info string, t *testing.T) {
61+
var dinf dockercompat.Info
62+
err := json.Unmarshal([]byte(stdout), &dinf)
63+
assert.NilError(t, err, "failed to parse docker info")
64+
for _, p := range dinf.Plugins.Storage {
65+
if p == "soci" {
66+
ret = true
67+
mess = "soci is enabled"
68+
}
69+
}
70+
},
71+
})
72+
73+
return ret, mess
74+
})
75+
76+
var Docker = test.MakeRequirement(func(data test.Data, t *testing.T) (ret bool, mess string) {
77+
ret = testutil.GetTarget() == testutil.Docker
78+
if ret {
79+
mess = "current target is docker"
80+
} else {
81+
mess = "current target is not docker"
82+
}
83+
return ret, mess
84+
})
85+
86+
var NerdctlNeedsFixing = test.MakeRequirement(func(data test.Data, t *testing.T) (ret bool, mess string) {
87+
ret = testutil.GetTarget() == testutil.Docker
88+
if ret {
89+
mess = "current target is docker"
90+
} else {
91+
mess = "current target is nerdctl, but it is currently broken and not working for this"
92+
}
93+
return ret, mess
94+
})
95+
96+
var Rootless = test.MakeRequirement(func(data test.Data, t *testing.T) (ret bool, mess string) {
97+
// Make sure we DO not return "IsRootless true" for docker
98+
ret = testutil.GetTarget() != testutil.Docker && rootlessutil.IsRootless()
99+
if ret {
100+
mess = "environment is rootless"
101+
} else {
102+
mess = "environment is rootful"
103+
}
104+
return ret, mess
105+
})
106+
107+
var Build = test.MakeRequirement(func(data test.Data, t *testing.T) (ret bool, mess string) {
108+
// FIXME: shouldn't we run buildkitd in a container? At least for testing, that would be so much easier than
109+
// against the host install
110+
ret = true
111+
mess = "buildkitd is enabled"
112+
if testutil.GetTarget() == testutil.Nerdctl {
113+
_, err := buildkitutil.GetBuildkitHost(testutil.Namespace)
114+
if err != nil {
115+
ret = false
116+
mess = fmt.Sprintf("buildkitd is not enabled: %+v", err)
117+
}
118+
}
119+
return ret, mess
120+
})
121+
122+
var CGroup = test.MakeRequirement(func(data test.Data, t *testing.T) (ret bool, mess string) {
123+
ret = true
124+
mess = "cgroup is enabled"
125+
(&test.GenericCommand{}).
126+
WithT(t).
127+
WithBinary(testutil.GetTarget()).
128+
WithArgs("info", "--format", "{{ json . }}").
129+
Run(&test.Expected{
130+
Output: func(stdout string, info string, t *testing.T) {
131+
var dinf dockercompat.Info
132+
err := json.Unmarshal([]byte(stdout), &dinf)
133+
assert.NilError(t, err, "failed to parse docker info")
134+
switch dinf.CgroupDriver {
135+
case "none", "":
136+
ret = false
137+
mess = "cgroup is none"
138+
}
139+
},
140+
})
141+
142+
return ret, mess
143+
})

0 commit comments

Comments
 (0)