|
1 | 1 | # ReTest |
2 | 2 |
|
3 | 3 | [](https://github.com/JuliaTesting/ReTest.jl/actions?query=workflow%3ACI) |
| 4 | +[](https://JuliaTesting.github.io/ReTest.jl/stable) |
| 5 | +[](https://JuliaTesting.github.io/ReTest.jl/dev) |
4 | 6 |
|
5 | | -This package allows: |
| 7 | +`ReTest` is a testing framework for Julia allowing: |
6 | 8 |
|
7 | | -1. Defining tests in source files, whose execution is deferred and triggered on demand. |
| 9 | +1. Defining tests in source files, whose execution is deferred and triggered |
| 10 | + on demand. |
8 | 11 |
|
9 | 12 | This is useful when one likes to have definitions of methods and |
10 | 13 | corresponding tests close to each other. This is also useful for code which |
11 | | - is not (yet) organized as a package, and where one doesn't want to maintain a |
12 | | - separate set of files for tests. |
| 14 | + is not (yet) organized as a package, and where one doesn't want to maintain |
| 15 | + a separate set of files for tests. |
13 | 16 |
|
14 | 17 | 2. Filtering run testsets with a `Regex`, which is matched against the |
15 | 18 | descriptions of testsets. |
16 | 19 |
|
17 | 20 | This is useful for running only part of the test suite of a package. For |
18 | | - example, if you made a change related to addition, and included "addition" in |
19 | | - the description of the corresponding testsets, you can easily run only these |
20 | | - tests. |
| 21 | + example, if you made a change related to addition, and included "addition" |
| 22 | + in the description of the corresponding testsets, you can easily run only |
| 23 | + these tests. |
21 | 24 |
|
22 | 25 | Note that a [pull request](https://github.com/JuliaLang/julia/pull/33672) |
23 | 26 | exists in the Julia repository to implement regex-filtering for |
24 | 27 | `Test.@testset`. |
25 | 28 |
|
26 | | -The exported `ReTest.@testset` macro can be used as a direct replacement for |
27 | | -`Test.@testset` (with limitations, see below), and `retest()` has to be called |
28 | | -for the tests to be executed. See the docstrings (reproduced below) for more |
29 | | -details. Moreover, `ReTest` re-exports (almost) all exported symbols from |
30 | | -`Test`, so there should not be any need to import `Test` together with `ReTest`. |
| 29 | +A couple more features are also enabled, like shuffling the order in which |
| 30 | +the testsets are run, or running testsets in parallel (via `Distributed`). |
31 | 31 |
|
32 | | -When using `@testset` "inline", i.e. within the source-code of a package, one |
33 | | -can use the `InlineTest` package instead of `ReTest`, which only defines the |
34 | | -strict minimum and also exports `@testset`, and therefore loads faster (even if |
35 | | -`ReTest` itself loads fast, it can be desirable to have an even lighter |
36 | | -dependency). But `ReTest` still has to be loaded (as a "test" dependency) in |
37 | | -order to call `retest`. |
38 | | - |
39 | | -Finally, for convenience, `@testset` also implicitly defines a `runtests` function |
40 | | -within the enclosing module, say `M`, such that `M.runtests(...)` is equivalent to |
41 | | -calling `retest(M, ...)`. |
42 | | - |
43 | | -### `retest` docstring |
44 | | - |
45 | | -``` |
46 | | - retest([m::Module], pattern = r""; dry::Bool=false, stats::Bool=false, |
47 | | - shuffle::Bool=false) |
48 | | -``` |
49 | | -Run all the tests declared in `@testset` blocks, within `m` if specified, |
50 | | -or within all currently loaded modules otherwise. |
51 | | -If `dry` is `true`, don't actually run the tests, just print the descriptions |
52 | | -of the testsets which would (presumably) run. |
53 | | -If `stats` is `true`, print some time/memory statistics for each testset. |
54 | | -If `shuffle` is `true`, shuffle the order in which top-level testsets are run. |
55 | | - |
56 | | -It's possible to filter run testsets by specifying `pattern`: the "subject" of a |
57 | | -testset is the concatenation of the subject of its parent `@testset`, if any, |
58 | | -with `"/$description"` where `description` is the testset's description. |
59 | | -For example: |
60 | | -```julia |
61 | | -@testset "a" begin # subject == "/a" |
62 | | - @testset "b" begin # subject is "/a/b" |
63 | | - end |
64 | | - @testset "c$i" for i=1:2 # subjects are "/a/c1" & "/a/c2" |
65 | | - end |
66 | | -end |
67 | | -``` |
68 | | -A testset is guaranteed to run only when its subject matches `pattern`. |
69 | | -Moreover if a testset is run, its enclosing testset, if any, also has to run |
70 | | -(although not necessarily exhaustively, i.e. other nested testsets |
71 | | -might be filtered out). |
72 | | - |
73 | | -If the passed `pattern` is a string, then it is wrapped in a `Regex` with the |
74 | | -"case-insensitive" flag, and must match literally the subjects. |
75 | | -This means for example that `"a|b"` will match a subject like `"a|b"` or `"A|B"`, |
76 | | -but not like `"a"` (only in Julia versions >= 1.3; in older versions, |
77 | | -the regex is simply created as `Regex(pattern, "i")`). |
78 | | - |
79 | | -Note: this function executes each (top-level) `@testset` block using `eval` *within* the |
80 | | -module in which it was written (e.g. `m`, when specified). |
81 | | - |
82 | | -### Caveats |
83 | | - |
84 | | -`ReTest.@testset` comes with a couple of caveats/limitations: |
85 | | - |
86 | | -* Toplevel testsets (which are not nested within other testsets), when run, are |
87 | | - `eval`ed at the toplevel of their parent module, which means that they can't |
88 | | - depend on local variables for example. This is probably the only fundamental |
89 | | - limitation compared to `Test.@testset`, and might not be fixable. |
90 | | - |
91 | | -* "testsets-for" (`@testset "description" for ...`), when run, imply |
92 | | - `eval`ing their loop variables at the toplevel of their parent module; |
93 | | - moreover, "testsets-for" accept only "non-cartesian" looping (e.g. `for i=I, |
94 | | - j=J` is not supported). Both problems should be fixable. |
95 | | - |
96 | | -* Testsets can not be "custom testsets" (cf. `Test` documentation; this should |
97 | | - be easy to support). |
98 | | - |
99 | | -* Nested testsets can't be "qualified" (i.e. written as `ReTest.@testset`). |
100 | | - |
101 | | -* Regex filtering logic might improve in future versions, which means that with |
102 | | - the same regex, less tests might be run (or more!). See `retest`'s docstring |
103 | | - to know what testsets are guaranteed to run. |
104 | | - |
105 | | -* Descriptions of testsets must be unique within a module, otherwise they are |
106 | | - overwritten and a warning is issued, unless `Revise` is loaded; |
107 | | - the reason is the current implemented heuristic to allow `Revise` do its magic. |
108 | | - |
109 | | -### Switching from `Test` to `ReTest` |
110 | | - |
111 | | -When used in a package `MyPackage`, the test code can be organized as follows: |
112 | | -1. replace `using Test` by `using ReTest` in the "runtests.jl" file |
113 | | -2. wrap the whole content of "runtests.jl" within a module named `MyPackageTests` |
114 | | -3. rename "runtests.jl" to "tests.jl" |
115 | | -4. create "runtests.jl" with the following content: |
116 | | - `include("tests.jl"); MyPackageTests.runtests()` |
117 | | - |
118 | | -This means that running "runtests.jl" will have the same net effect as before. |
119 | | -The "tests.jl" file can now be `include`d in your REPL session (`include("tests.jl")`), |
120 | | -and you can run all or some of its tests |
121 | | -(e.g. `MyPackageTests.runtests("addition")`). |
122 | | -Wrapping the tests in `MyPackageTests` allows to not pollute `Main`, it keeps the tests |
123 | | -of different packages separated, but more importantly, you can modify "tests.jl" and |
124 | | -re-include it to have the corresponding tests updated (otherwise, without |
125 | | -a `MyPackageTests` module, including the file a second time would add the new tests |
126 | | -without removing the old ones). |
127 | | - |
128 | | -### Filtering |
129 | | - |
130 | | -Most of the time, filtering with a simple string is likely to be enough. For example, in |
131 | | -```julia |
132 | | -@testset "a" begin |
133 | | - @test true |
134 | | - @testset "b" begin |
135 | | - end |
136 | | - @testset "c" begin |
137 | | - end |
138 | | -end |
139 | | -``` |
140 | | - |
141 | | -running `retest(M, "a")` will run everything, `retest(M, "b")` will run |
142 | | -`@test true` and `@testset "b"` but not `@testset "c"`. |
143 | | -Note that if you want to run `@testset "b"`, there is no way to not run |
144 | | -`@test true` in `@testset "a"`; so if it was an expensive test to run, |
145 | | -instead of `@test true`, it could be useful to wrap it in its own testset, so that |
146 | | -it can be filtered out. |
147 | | - |
148 | | -### Working with `Revise` |
149 | | - |
150 | | -When `Revise` is loaded and a testset is updated, `ReTest` will observe that a |
151 | | -new testset is added with the same description as a previously existing one, |
152 | | -which is then overwritten. This works only if the description is not modified, |
153 | | -otherwise both the old and new versions of the testset will co-exist. |
154 | | - |
155 | | -For testsets in a "script" loaded with `includet`, e.g. those in a |
156 | | -"test/tests.jl" file, you can request `Revise` to "load" the updated testsets by |
157 | | -putting `__revise_mode__ = :eval` in the enclosing module. |
158 | | - |
159 | | -When files are included recursively, plain `includet` won't work |
160 | | -(it is currently documented to be "deliberately non-recursive"). |
161 | | -There are two work-arounds: |
162 | | -1. rename your "test/tests.jl" file to "test/MyPackageTests.jl" and load it as a module |
163 | | - (this might involve updating your `LOAD_PATH` to include "test/" and making sure |
164 | | - the required packages are found) |
165 | | -2. use the [following `recursive_includet`](https://github.com/timholy/Revise.jl/issues/518#issuecomment-667097500) |
166 | | - function instead of `includet`: |
167 | | -```julia |
168 | | -function recursive_includet(filename) |
169 | | - already_included = copy(Revise.included_files) |
170 | | - includet(filename) |
171 | | - newly_included = setdiff(Revise.included_files, already_included) |
172 | | - for (mod, file) in newly_included |
173 | | - Revise.track(mod, file) |
174 | | - end |
175 | | -end |
176 | | -``` |
177 | | - |
178 | | -### Working with test files which use Test |
179 | | - |
180 | | -It's sometimes possible to use `ReTest` features on a test code base which uses `Test`: |
181 | | -- if you have a package `Package`, you can try `ReTest.hijack(Package)`, which will define |
182 | | -a `PackageTests` module when successful, on which you can call `retest`. |
183 | | -- if you have a test file `"testfile.jl"`, try `ReTest.hijack("testfile.jl")` |
184 | | -(this will define a fresh module like above). |
185 | | - |
186 | | -Check out the docstring of `ReTest.hijack` for more details. |
| 32 | +`ReTest` is mostly backward-compatible with `Test`, i.e. minimal change to |
| 33 | +test files is necessary in order to switch to `ReTest`; it's often even |
| 34 | +possible to use `ReTest` features without changing a line, e.g. on Julia's |
| 35 | +`Base`/stdlib tests. |
0 commit comments