Skip to content

Commit 15d291f

Browse files
rfourquetrfourquet
committed
update README
1 parent f1a7251 commit 15d291f

File tree

1 file changed

+176
-12
lines changed

1 file changed

+176
-12
lines changed

‎README.md

Lines changed: 176 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,178 @@
11
# InlineTest
22

3-
This package allows defining tests in source files, whose execution is deferred and triggered on demand.
4-
This is useful when one likes to have definitions of methods and corresponding tests close to each other.
5-
This is also useful for code which is not (yet) organized as a package, and where one doesn't want to maintain
6-
a separate set of files for tests.
7-
8-
The exported `InlineTest.@testset` macro can be used as a direct replacement for `Test.@testset`,
9-
and `runtests()` has to be called for the tests to be executed. See the docstrings for more details.
10-
When used in a package `MyPackage`, the following can be added to its `test/runtests.jl` file
11-
in order to have `InlineTest.@testset` tests run as part of the usual package testing process:
12-
`runtests(MyPackage)` (depending on whether `Test` is imported in the test file -- in which case
13-
there would be a name resolution conflict for `@testset` -- you might bring
14-
`runtests` in scope either with `using InlineTest` or `using InlineTest: runtests`).
3+
This package allows:
4+
5+
1. defining tests in source files, whose execution is deferred and triggered on demand.
6+
7+
This is useful when one likes to have definitions of methods and
8+
corresponding tests close to each other. This is also useful for code which
9+
is not (yet) organized as a package, and where one doesn't want to maintain a
10+
separate set of files for tests.
11+
12+
2. filtering run testsets with a `Regex`, which is matched against the
13+
descriptions of testsets.
14+
15+
This is useful for running only part of the test suite of a package. For
16+
example, if you made a change related to addition, and included "addition" in
17+
the description of the corresponding tetstsets, you can easily run only these
18+
tests.
19+
20+
Note that a [pull request](https://github.com/JuliaLang/julia/pull/33672)
21+
exists in the Julia repository to implement regex-filtering for
22+
`Test.@testset`.
23+
24+
The exported `InlineTest.@testset` macro can be used as a direct replacement for
25+
`Test.@testset`, and `runtests()` has to be called for the tests to be executed.
26+
See the docstrings (reproduced below) for more details. Moreover, `InlineTest`
27+
re-exports (almost) all exported symbols from `Test`, so there should not be any
28+
need to import `Test` together with `InlineTest`.
29+
30+
### `runtests` docstring
31+
32+
```
33+
runtests([m::Module], pattern = r""; [wrap::Bool])
34+
```
35+
Run all the tests declared in `@testset` blocks, within `m` if specified,
36+
or within all currently loaded modules otherwise.
37+
The `wrap` keyword specifies whether the collection of `@testset` expressions
38+
should be grouped according to the parent modules within a top-level `@testset`.
39+
The default is `wrap=false` when `m` is specified, `true` otherwise.
40+
41+
It's possible to filter run testsets by specifying `pattern`: the "subject" of a
42+
testset is the concatenation of the subject of its parent `@testset`, if any,
43+
with `"/$description"` where `description` is the testset's description.
44+
For example:
45+
```julia
46+
@testset "a" begin # subject == "/a"
47+
@testset "b" begin # subject is "/a/b"
48+
end
49+
@testset "c$i" for i=1:2 # subjects are "/a/c1" & "/a/c2"
50+
end
51+
end
52+
```
53+
A testset is guaranteed to run only when its parent's subject "partially matches"
54+
`pattern::Regex` (in PCRE2 parlance, i.e. `subject` might be the prefix of a string
55+
matched by `pattern`) and its subject matches `pattern`.
56+
57+
If the passed `pattern` is a string, then it is wrapped in a `Regex` prefixed with
58+
`".*"`, and must match literally the subjects.
59+
This means for example that `"a|b"` will match a subject like `"a|b"` but not like `"a"`.
60+
The `".*"` prefix is intended to allow matching subjects of nested testsets,
61+
e.g. in the example above, `r".*b"` partially matches the subject `"/a"` and
62+
matches the subject `"/a/b"` (so the corresponding nested testset is run),
63+
whereas `r"b"` does not partially match `"/a"`, even if it matches `"/a/b"`,
64+
so the testset is not run.
65+
66+
Note: this function executes each (top-level) `@testset` block using `eval` *within* the
67+
module in which it was written (e.g. `m`, when specified).
68+
69+
### Caveats
70+
71+
`InlineTest.@testset` comes with a couple of caveats/limitations:
72+
73+
* Toplevel testsets (which are not nested within other testsets), when run, are
74+
`eval`ed at the toplevel of their parent module, which means that they can't
75+
depend on local variables for example. This is probably the only fundamental
76+
limitation compared to `Test.@testset`, and might not be fixable.
77+
78+
* Toplevel "testsets-for" (`@testset "description" for ...`), when run, imply
79+
`eval`ing their loop variables at the toplevel of their parent module;
80+
moreover, "testsets-for" accept only "non-cartesian" looping (e.g. `for i=I,
81+
j=J` is not supported). Both problems should be fixable.
82+
83+
Related to the previous point: in future versions, nested testsets-for might
84+
have their "iterator" (giving the values their loop variables) `eval`ed at the
85+
module's toplevel (for efficiency).
86+
87+
* Testsets can not be "custom testsets" (cf. `Test` documentation; this should
88+
be easy to support).
89+
90+
* Nested testsets can't be "qualified" (i.e. written as `InlineTest.@testset`).
91+
92+
* Regex filtering logic might improve in future versions, which means that with
93+
the same regex, less tests might be run (or more!). See `runtests`'s docstring
94+
to know what testsets are guaranteed to run.
95+
96+
### Switching from `Test` to `InlineTest`
97+
98+
When used in a package `MyPackage`, the test code can be organized as follows:
99+
1. replace `using Test` by `using InlineTest` in the "runtests.jl" file
100+
2. wrap the whole content of "runtests.jl" within a module named `MyPackageTests`
101+
3. rename "runtests.jl" to "tests.jl"
102+
4. create "runtests.jl" with the following content:
103+
`include("tests.jl"); MyPackageTests.runtests(MyPackageTests)`
104+
105+
This means that running "runtests.jl" will have the same net effect as before.
106+
The "tests.jl" file can now be `include`d in your REPL session (`include("tests.jl")`),
107+
and you can run all or some of its tests
108+
(e.g. `using InlineTest; runtests(MyPackageTests, "addition")`).
109+
Wrapping the tests in `MyPackageTests` allows to not pollute `Main`, it keeps the tests
110+
of different packages separated, but more importantly, you can modify "tests.jl" and
111+
re-include it to have the corresponding tests updated (otherwise, without
112+
a `MyPackageTests` module, including the file a second time would add the new tests
113+
without removing the old ones).
114+
115+
### Toplevel testsets
116+
117+
In `InlineTest`, toplevel testsets are special, as already mentioned. The fact
118+
that they are `eval`ed at the module's toplevel also means that we can actually
119+
filter them out (if a filtering pattern is given to `runtests`) "statically",
120+
i.e. by introspection, we can figure out whether they should be run before
121+
having to `eval` the corresponding code. This is a big win, in particular for
122+
`testsets-for`, which are expensive to compile. This allows `InlineTest` to
123+
compete with the good-old manual way of copy-pasting the wanted `@testset` into
124+
the REPL (without this trick, all the testsets would have to be `eval`ed, even
125+
when they don't run any code, and this can take some time for large test
126+
codebases.
127+
128+
So the recommendation for efficient filtering of the tests is to favor toplevel
129+
testsets, e.g.
130+
131+
```julia
132+
@testset "a" begin
133+
# ...
134+
end
135+
@testset "b" begin
136+
# ...
137+
end
138+
# etc.
139+
```
140+
instead of
141+
```julia
142+
@testset "everything" begin
143+
@testset "a" begin
144+
# ...
145+
end
146+
@testset "b" begin
147+
# ...
148+
end
149+
# etc.
150+
end
151+
```
152+
153+
### Filtering
154+
155+
Most of the time, filtering with a simple string is likely to be enough. For example, in
156+
```julia
157+
@testset "a" begin
158+
@test true
159+
@testset "b" begin
160+
end
161+
@testset "c" begin
162+
end
163+
end
164+
```
165+
166+
running `runtests(M, "a")` will run everything, `runtests(M, "b")` will run
167+
`@test true` and `@testset "b"` but not `@testset "c"`. Using `Regex` is
168+
slightly more subtle. For example, `runtests(M, r"a")` is equivalent to
169+
`runtests(M, "a")` in this case, but `runtests(M, r"b")` will run nothing,
170+
because `r"b"` doesn't "partially match" the toplevel `@testset "a"`; you would
171+
have to use `r".*b"` instead for example, or `r"a/b"` (or even `r"/b"`).
172+
173+
Note also that partial matching often gives a false positive, e.g. running
174+
`runtests(M, "d")` will currently instantiate `@testset "a"` because it might
175+
contain a sub-testset `"d"`, so `@test true` above will be run, even if
176+
eventually no testset matches `"d"`. So it's recommended to put expensive tests
177+
withing "final" testsets (those which don't have nested testsets), such that
178+
"full matching" is used instead of partial matching.

0 commit comments

Comments
 (0)