|
1 | 1 | # How does Descartes work? |
2 | 2 |
|
3 | | -Descartes finds the worst tested methods in a project by using *mutation testing*, a well known technique to assess the quality of a test suite by introducing artificial faults and then checking if the tests are able to detect them. In particular the tool uses *extreme mutation testing* that works at the method level. |
| 3 | +Descartes finds the worst tested methods in a project by using *mutation testing*. This is a well known technique to assess the quality of a test suite by introducing artificial faults and then checking if the tests are able to detect them. In particular the tool uses *extreme mutation testing* that works at the method level. |
4 | 4 |
|
5 | 5 | ## Mutation testing |
6 | 6 | Mutation testing verifies if your test suite can detect bugs. |
7 | 7 | The technique works by introducing small changes or faults into the original program. These modified versions are called **mutants**. |
8 | 8 | A good test suite should be able to detect or *kill* a mutant. That is, at least one test case should fail when the test suite is executed with the mutant instead of the original program. |
9 | 9 | [Read more](https://en.wikipedia.org/wiki/Mutation_testing). |
10 | | -Traditional mutation testing works at the instruction level, e.g., replacing ">" by "<=", so the number of generated mutants is huge, and it takes more time to check the entire test suite. |
| 10 | +Traditional mutation testing works at the instruction level, e.g., replacing ">" by "<=", so the number of generated mutants is huge, and it takes a lot of time to check the entire test suite. |
11 | 11 | That's why the authors of [Will my tests tell me if I break this code?](http://dl.acm.org/citation.cfm?doid=2896941.2896944) proposed an *Extreme Mutation* strategy, which works at the method level. |
12 | 12 |
|
13 | 13 | ## Extreme Mutation Testing |
14 | 14 | In Extreme Mutation testing, instead of changing one instruction at a time, the whole logic of a method under test is eliminated. |
15 | | -All statements in a `void` method are removed. If he method is not `void`, the body is replaced by a single return statement producing a constant value. |
| 15 | +All statements in a `void` method are removed. If the method is not `void`, the body is replaced by a single return statement producing a constant value. |
16 | 16 | This approach generates fewer mutants. Code from the authors can be found in [their GitHub repository](https://github.com/cqse/test-analyzer). |
17 | 17 |
|
18 | | -The goal of Descartes is to bring an effective implementation of this kind of mutation operator into the world of [PIT](https://pitest.org) and check its performance in real world projects. |
| 18 | +The goal of Descartes is to bring an effective implementation of this kind of mutation operator into the world of [PIT](https://pitest.org) and check its performance and effectiveness in real world projects. |
19 | 19 |
|
20 | 20 | ## Method classification |
21 | 21 | Extreme mutation classify methods according to the outcome of the extreme mutants. If no extreme mutant created for a method are detected by the test suite it is then considered as **pseudo-tested**. These are the worst tested methods in the project. If there are mixed results, that is, for the same method some extreme mutants are detected and others are not, then the method is classified as **partially-tested**. These are not well tested either. Otherwise, if all mutants are detected the method is considered as **tested**. |
22 | 22 |
|
23 | 23 | Descartes finds and reports all **pseudo-tested** and **partially-tested** methods in a given project. |
24 | 24 |
|
25 | | -## Mutation operators |
26 | | -The *mutation operators* are models of the artificial faults used to create mutants. In particular, *extreme mutation operators* replace the body of a method by one simple return instruction or just remove all instructions if possible. The set of mutation operators that Descartes uses can be configured. Further details on how to configure are given in ["Running Descartes on your project"](./running-on-your-project.html). You can find frequent configuration examples [here](./frequent-configurations.html). |
27 | | - |
28 | | -Below you can find the full list of extreme mutation operators used by Descartes. |
29 | | - |
30 | | -## `void` mutation operator |
31 | | -This operator accepts a `void` method and removes all the instructions on its body. For example, with the following class as input: |
32 | | - |
33 | | -```java |
34 | | -class A { |
35 | | - |
36 | | - int field = 3; |
37 | | - |
38 | | - public void Method(int inc) { |
39 | | - field += 3; |
40 | | - } |
41 | | - |
42 | | -} |
43 | | -``` |
44 | | -the mutation operator will generate: |
45 | | - |
46 | | -```java |
47 | | -class A { |
48 | | - |
49 | | - int field = 3; |
50 | | - |
51 | | - public void Method(int inc) { } |
52 | | - |
53 | | -} |
54 | | -``` |
55 | | - |
56 | | -## `null` mutation operator |
57 | | -This operator accepts a method with a reference return type and replaces all instructions |
58 | | -with `return null`. For example, using the following class as input: |
59 | | -```java |
60 | | -class A { |
61 | | - public B create() { |
62 | | - return new B(); |
63 | | - } |
64 | | -} |
65 | | -``` |
66 | | -this operator will generate: |
67 | | - |
68 | | -```java |
69 | | -class A { |
70 | | - public B create() { |
71 | | - return null; |
72 | | - } |
73 | | -} |
74 | | -``` |
75 | | -## `empty` mutation operator |
76 | | -This is a special operator which targets methods that return arrays. It replaces the entire |
77 | | -body with a `return` statement that produces an empty array of the corresponding type. |
78 | | -For example, the following class: |
79 | | - |
80 | | -```java |
81 | | -class A { |
82 | | - public int[] getRange(int count) { |
83 | | - int[] result = new int[count]; |
84 | | - for(int i=0; i < count; i++) { |
85 | | - result[i] = i; |
86 | | - } |
87 | | - return result; |
88 | | - } |
89 | | -} |
90 | | -``` |
91 | | -will become: |
92 | | - |
93 | | -```java |
94 | | -class A { |
95 | | - public int[] getRange(int count) { |
96 | | - return new int[0]; |
97 | | - } |
98 | | -} |
99 | | -``` |
100 | | - |
101 | | -## Constant mutation operator |
102 | | -This operator accepts any method with a primitive or `String` return type. It replaces the method body |
103 | | -with a single instruction returning a defined constant. |
104 | | -For example, if the integer constant `3` is specified, then for the following class: |
105 | | - |
106 | | -```java |
107 | | -class A { |
108 | | - int field; |
109 | | - |
110 | | - public int getAbsField() { |
111 | | - if(field >= 0) |
112 | | - return field; |
113 | | - return -field; |
114 | | - } |
115 | | -} |
116 | | -``` |
117 | | -this operator will generate: |
118 | | - |
119 | | -```java |
120 | | -class A { |
121 | | - int field; |
122 | | - |
123 | | - public int getAbsField() { |
124 | | - return 3; |
125 | | - } |
126 | | -} |
127 | | -``` |
128 | | - |
129 | | -## `new` mutation operator |
130 | | -*New in version 1.2.6* |
131 | | - |
132 | | -This operator accepts any method whose return type has a constructor with no parameters and belongs to a `java` package. |
133 | | -It replaces the code of the method by a single instruction returning a new instance. |
134 | | - |
135 | | -For example: |
136 | | - |
137 | | -```java |
138 | | -class A { |
139 | | - int field; |
140 | | - |
141 | | - public ArrayList range(int end) { |
142 | | - ArrayList l = new ArrayList(); |
143 | | - for(int i = 0; i < size; i++) { |
144 | | - A a = new A(); |
145 | | - a.field = i; |
146 | | - l.add(a); |
147 | | - } |
148 | | - return l; |
149 | | - } |
150 | | -} |
151 | | -``` |
152 | | - |
153 | | -is transformed to: |
154 | | - |
155 | | -```java |
156 | | -class A { |
157 | | - int field; |
158 | | - |
159 | | - public List range(int end) { |
160 | | - return new ArrayList(); |
161 | | - } |
162 | | -} |
163 | | -``` |
164 | | - |
165 | | -This operator handles the following special cases: |
166 | | - |
167 | | -| Return Type | Replacement | |
168 | | -|--------------|--------------| |
169 | | -| `Collection` | `ArrayList` | |
170 | | -| `Iterable` | `ArrayList` | |
171 | | -| `List` | `ArrayList` | |
172 | | -| `Queue` | `LinkedList` | |
173 | | -| `Set` | `HashSet` | |
174 | | -| `Map` | `HashMap` | |
175 | | - |
176 | | -This means that if a method returns an instance of `Collection` the code of the mutated method will be |
177 | | -`return new ArrayList();`. |
178 | | - |
179 | | -This operator is not enabled by default. |
180 | | - |
181 | | -## `optional` mutation operator |
182 | | -*New in version 1.2.6* |
183 | | - |
184 | | -This operator accepts any method whose return type is `java.util.Optional`. |
185 | | -It replaces the code of the method by a single instruction returning an *empty* instance. |
186 | | - |
187 | | -For example: |
188 | | - |
189 | | -```java |
190 | | -class A { |
191 | | - int field; |
192 | | - |
193 | | - public Optional<Integer> getOptional() { |
194 | | - return Optional.of(field); |
195 | | - } |
196 | | -} |
197 | | -``` |
198 | | - |
199 | | -is transformed to: |
200 | | - |
201 | | -```java |
202 | | -class A { |
203 | | - int field; |
204 | | - |
205 | | - public Optional<Integer> getOptional() { |
206 | | - return Optional.empty(); |
207 | | - } |
208 | | -} |
209 | | -``` |
210 | | - |
211 | | -This operator is not enabled by default. |
212 | | - |
213 | | -## `argument` mutation operator |
214 | | -*New in version 1.3* |
215 | | - |
216 | | -This operator replaces the body of a method by returning the value of the first parameter that has the same type as the return type of the method. |
217 | | - |
218 | | -For example: |
219 | | - |
220 | | -```java |
221 | | -class A { |
222 | | - public int m(int x, int y) { |
223 | | - return x + 2 * y; |
224 | | - } |
225 | | -} |
226 | | -``` |
227 | | - |
228 | | -is transformed to: |
229 | | - |
230 | | -```java |
231 | | -class A { |
232 | | - public int m(int x) { |
233 | | - return x; |
234 | | - } |
235 | | -} |
236 | | -``` |
237 | | - |
238 | | -This operator is not enabled by default. |
239 | | - |
240 | | -## `this` mutation operator |
241 | | -*New in version 1.3* |
242 | | - |
243 | | -Replaces the body of a method by `return this;` if applicable. The goal of this operator is to perform better transformations targeting fluent APIs. |
244 | | - |
245 | | -For example: |
246 | | - |
247 | | -```java |
248 | | -class A { |
249 | | - |
250 | | - int value = 0; |
251 | | - public A addOne() { |
252 | | - value += 1; |
253 | | - return this; |
254 | | - } |
255 | | -} |
256 | | -``` |
257 | | - |
258 | | -is transformed to: |
259 | | - |
260 | | -```java |
261 | | -class A { |
262 | | - |
263 | | - int value = 0; |
264 | | - public A addOne() { |
265 | | - return this; |
266 | | - } |
267 | | -} |
268 | | -``` |
269 | | - |
270 | | -This operator is not enabled by default. |
271 | | - |
272 | | -## Stop Methods |
273 | | - |
274 | | -Descartes avoids some methods that are generally not interesting and may introduce false positives such as simple getters, simple setters, empty void methods or methods returning constant values, delegation patterns as well as deprecated and compiler generated methods. Those methods are automatically detected by inspecting their code. |
275 | | -A complete list of examples can be found [here](https://github.com/STAMP-project/pitest-descartes/blob/master/src/test/java/eu/stamp_project/test/input/StopMethods.java). |
276 | | -The exclusion of stop methods can be configured. For more details see: ["Running Descartes on your project"](./running-on-your-project.html). |
277 | | - |
278 | | -## Do not use `null` in methods annotated with `@NotNull` |
279 | | -*New in version 1.2.6* |
280 | | - |
281 | | -Descartes will avoid using the `null` operator in methods annotated with `@NotNull`. This increases its compatibility with Kotlin sources. This feature can be configured. See ["Running Descartes on your project"](./running-on-your-project.html) for more details. |
282 | | - |
283 | | -## Skip methods using `@DoNotMutate` |
284 | | -*New in version 1.3* |
285 | | - |
286 | | -Descartes skips all method that are annotated with *any* annotation whose name is `DoNotMutate`. |
287 | | -For example, in the following fragment of code, the method `m` will not be mutated. |
288 | | - |
289 | | -```java |
290 | | -class A { |
291 | | - @DoNotMutate |
292 | | - public void m() {/* ... */} |
293 | | -} |
294 | | -``` |
295 | | - |
296 | | -All methods in a class annotated with `@DoNotMutate` will be avoided as well. For example, in the following fragment of code, the method `m` will not be mutated: |
297 | | - |
298 | | -```java |
299 | | -@DoNotMutate |
300 | | -class A { |
301 | | - public void m() {/* ... */} |
302 | | -} |
303 | | -``` |
304 | | - |
305 | | -The `DoNotMutate` annotation may specify which operators should be considered. For example: |
306 | | - |
307 | | -```java |
308 | | -class A { |
309 | | - @DoNotMutate(operators = "false") |
310 | | - public boolean m() { return true; } |
311 | | -} |
312 | | -``` |
313 | | - |
314 | | -will instruct Descartes not to use the `false` mutation operator to mutate `m`. |
315 | | - |
316 | | -When specifying operators, a method annotation takes precedence over class annotations. That, is the `@DoNotMutate` of a method overrides the same annotation in the class For example: |
317 | | - |
318 | | -```java |
319 | | -@DoNotMutate(operators = "true") |
320 | | -class A { |
321 | | - |
322 | | - public boolean n() { return false; } |
323 | | - |
324 | | - @DoNotMutate(operators = "false") |
325 | | - public boolean m() { return true; } |
326 | | -} |
327 | | -``` |
328 | | - |
329 | | -will not mutate method `n` with `true`, as instructed in the class annotation. On the other hand, `m` will be mutated by `true` but not by `false`. |
330 | | - |
331 | | -Descartes includes a definition of `DoNotMutate`. However, when the tool inspects the annotations of a class or method it matches only the simple name of the annotation class and ignores the package. So, any `DoNotMutate` annotation will be considered. In this way a project does not need to add Descartes as a dependency, it can declare its own `DoNotMutate` and use it. |
332 | | - |
333 | | -This feature is also configurable. See ["Running Descartes on your project"](./running-on-your-project.html) for more details. |
| 25 | +Descartes uses a set of transformation models or *extreme mutation operators* to analyse each method. The full list and details of these mutation operators can be checked [here](./mutation-operators.html). |
0 commit comments