You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This vignette is a quick reference guide for testing challenging functions. It's organised by the problem, rather than the technique used to solve it, so you can quickly skim the whole vignette, spot the problem you're facing, then learn more about useful tools for solving it.
23
+
This vignette is a quick reference guide for testing challenging functions. It's organized by the problem, rather than the technique used to solve it, so you can quickly skim the whole vignette, spot the problem you're facing, then learn more about useful tools for solving it.
You can skip a test without passing or failing if it's not possible to run it in the current environment (e.g. it's OS dependent, or it only works interactively, or it shouldn't be tested on CRAN). Learn more in `vignette("skipping")`.
66
+
You can skip a test without passing or failing if it's not possible to run it in the current environment (e.g., it's OS dependent, or it only works interactively, or it shouldn't be tested on CRAN). Learn more in `vignette("skipping")`.
67
67
68
68
## HTTP requests
69
69
@@ -145,4 +145,4 @@ Learn more in `vignette("mocking")`.
145
145
146
146
## User-facing text
147
147
148
-
Errors, warnings, and other user-facing text should be tested to ensure they're consistent and actionable. Obviously you can't test this 100% automatically, but you can ensure that such messaging is clearly shown in PRs so another human can take a look. This is the point of snapshot tests; learn more in `vignette("snapshotting")`.
148
+
Errors, warnings, and other user-facing text should be tested to ensure they're consistent and actionable. Obviously, you can't test this 100% automatically, but you can ensure that such messaging is clearly shown in PRs so another human can take a look. This is the point of snapshot tests; learn more in `vignette("snapshotting")`.
The first step in any expectation is to use `quasi_label()` to capture a "labelled value", i.e. a list that contains both the value (`$val`) for testing and a label (`$lab`) for messaging. This is a pattern that exists for fairly esoteric reasons; you don't need to understand it, just copy and paste it 🙂.
46
+
The first step in any expectation is to use `quasi_label()` to capture a "labelled value", i.e., a list that contains both the value (`$val`) for testing and a label (`$lab`) for messaging. This is a pattern that exists for fairly esoteric reasons; you don't need to understand it, just copy and paste it 🙂.
47
47
48
48
Next you need to check each way that `object` could violate the expectation. In this case, there's only one check, but in more complicated cases there can be multiple checks. In most cases, it's easier to check for violations one by one, using early returns to `fail()`. This makes it easier to write informative failure messages that first describe what was expected and then what was actually seen.
49
49
@@ -87,7 +87,7 @@ The following sections show you a few more variations, loosely based on existing
87
87
88
88
### `expect_vector_length()`
89
89
90
-
Let's make `expect_length()` a bit more strict by also checking that the input is a vector. R is a bit weird in that it gives a length to pretty much every object, and you can imagine not wanting this code to succeed:
90
+
Let's make `expect_length()` a bit more strict by also checking that the input is a vector. R is a bit unusual in that it gives a length to pretty much every object, and you can imagine not wanting this code to succeed:
91
91
92
92
```{r}
93
93
expect_length(mean, 1)
@@ -182,7 +182,7 @@ Note the variety of messages:
182
182
183
183
* When `object` isn't an object, we only need to say what we expect.
184
184
* When `object` isn't an S3 object, we know it's an S4 object.
185
-
* When `inherits()` is `FALSE`, we provide the actual _class_, since that's most informative.
185
+
* When `inherits()` is `FALSE`, we provide the actual *class*, since that's most informative.
186
186
187
187
I also check that the `class` argument must be a string. This is an error, not a failure, because it suggests you're using the function incorrectly.
Mocking allows you to temporarily replace the implementation of a function with something that makes it easier to test. It's useful when testing failure scenarios that are hard to generate organically (e.g. what happens if dependency X isn't installed?), making tests more reliable by eliminating potential variability, and making tests faster. It's also a general escape hatch to resolve pretty much any challenging testing problem.
23
+
Mocking allows you to temporarily replace the implementation of a function with something that makes it easier to test. It's useful when testing failure scenarios that are hard to generate organically (e.g., what happens if dependency X isn't installed?), making tests more reliable by eliminating potential variability, and making tests faster. It's also a general escape hatch to resolve pretty much any challenging testing problem.
24
24
25
-
(If, like me, you're confused as to why you'd want to cruelly make fun of your tests, here mocking is used in the sense of making a fake or simulated version of something, i.e. a mock-up.)
25
+
(If, like me, you're confused as to why you'd want to cruelly make fun of your tests, here mocking is used in the sense of making a fake or simulated version of something, i.e., a mock-up.)
26
26
27
-
testthat supports mocking primarily through `local_mocked_bindings()` for mocking functions, and we'll focus on that function in this vignette. But it also provides other methods for specialised cases: you can use `local_mocked_s3_method()` to mock an S3 method, `local_mocked_s4_method()` to mock a S4 method, and `local_mocked_r6_class()` to mock an R6 class. Once you understand the basic idea of mocking, I think it should be straightforward to apply these other functions where needed.
27
+
testthat supports mocking primarily through `local_mocked_bindings()` for mocking functions, and we'll focus on that function in this vignette. But it also provides other methods for specialized cases: you can use `local_mocked_s3_method()` to mock an S3 method, `local_mocked_s4_method()` to mock an S4 method, and `local_mocked_r6_class()` to mock an R6 class. Once you understand the basic idea of mocking, I think it should be straightforward to apply these other functions where needed.
28
28
29
29
## Getting started with mocking
30
30
31
-
Let's begin by motivating mocking with a simple example. Imagine you're writing a function like `rlang::check_installed()`. The goal of this function is to check if a package is installed, and if not, give a nice error message. It also takes an option`min_version` argument which you can use to enforce a version constraint. A simple base R implementation might look something like this:
31
+
Let's begin by motivating mocking with a simple example. Imagine you're writing a function like `rlang::check_installed()`. The goal of this function is to check if a package is installed, and if not, give a nice error message. It also takes an optional`min_version` argument which you can use to enforce a version constraint. A simple base R implementation might look something like this:
But it's starting to feel like we've accumulated a bunch of potentially fragile hacks. So let's see how we could make these tests more robust with mocking. First we need to add `requireNamespace` and `packageVersion` bindings in our package. This is needed because `requireNamespace` and `packageVersion` are base functions:
87
+
But it's starting to feel like we've accumulated a bunch of potentially fragile hacks. So let's see how we could make these tests more robust with mocking. First, we need to add `requireNamespace` and `packageVersion` bindings in our package. This is needed because `requireNamespace` and `packageVersion` are base functions:
88
88
```{r}
89
89
requireNamespace <- NULL
90
90
packageVersion <- NULL
91
91
```
92
92
93
-
For the first test, we mock `requireNamespace()` twice, first returning `TRUE`, pretending every package is installed, and then returning `FALSE` pretending that no packages are installed.
93
+
For the first test, we mock `requireNamespace()` twice, first returning `TRUE`, pretending every package is installed, and then returning `FALSE`, pretending that no packages are installed.
94
94
95
95
```{r}
96
96
test_that("check_installed() checks package is installed", {
Copy file name to clipboardExpand all lines: vignettes/skipping.Rmd
+6-6Lines changed: 6 additions & 6 deletions
Original file line number
Diff line number
Diff line change
@@ -20,14 +20,14 @@ Skipping is a relatively advanced topic because in most cases you want all your
20
20
The most common exceptions are:
21
21
22
22
- You're testing a web service that occasionally fails, and you don't want to run the tests on CRAN.
23
-
Or maybe the API requires authentication, and you can only run the tests when you've [securely distributed](https://gargle.r-lib.org/articles/articles/managing-tokens-securely.html) some secrets.
23
+
Or the API requires authentication, and you can only run the tests when you've [securely distributed](https://gargle.r-lib.org/articles/articles/managing-tokens-securely.html) some secrets.
24
24
25
25
- You're relying on features that not all operating systems possess, and want to make sure your code doesn't run on a platform where it doesn't work.
26
-
This platform tends to be Windows, since amongst other things, it lacks full utf8 support.
26
+
This platform tends to be Windows, since among other things, it lacks full UTF-8 support.
27
27
28
28
- You're writing your tests for multiple versions of R or multiple versions of a dependency and you want to skip when a feature isn't available.
29
29
You generally don't need to skip tests if a suggested package is not installed.
30
-
This is only needed in exceptional circumstances, e.g. when a package is not available on some operating system.
30
+
This is only needed in exceptional circumstances, e.g., when a package is not available on some operating system.
31
31
32
32
```{r setup}
33
33
library(testthat)
@@ -43,12 +43,12 @@ testthat comes with a variety of helpers for the most common situations:
43
43
-`skip_on_os()` allows you to skip tests on a specific operating system.
44
44
Generally, you should strive to avoid this as much as possible (so your code works the same on all platforms), but sometimes it's just not possible.
45
45
46
-
-`skip_on_ci()` skips tests on most continuous integration platforms (e.g. GitHub Actions, Travis, Appveyor).
46
+
-`skip_on_ci()` skips tests on most continuous integration platforms (e.g., GitHub Actions, Travis, Appveyor).
47
47
48
48
You can also easily implement your own using either `skip_if()` or `skip_if_not()`, which both take an expression that should yield a single `TRUE` or `FALSE`.
49
49
50
50
All reporters show which tests are skipped.
51
-
As of testthat 3.0.0, ProgressReporter (used interactively) and CheckReporter (used inside of `R CMD check`) also display a summary of skips across all tests.
51
+
As of testthat 3.0.0, ProgressReporter (used interactively) and CheckReporter (used inside `R CMD check`) also display a summary of skips across all tests.
Another potentially useful technique is to build a `skip()` directly into a package function.
80
-
For example, take a look at [`pkgdown:::convert_markdown_to_html()`](https://github.com/r-lib/pkgdown/blob/v2.0.7/R/markdown.R#L95-L106), which absolutely, positively cannot work if the Pandoc tool is unavailable:
80
+
For example, take a look at [`pkgdown:::convert_markdown_to_html()`](https://github.com/r-lib/pkgdown/blob/v2.0.7/R/markdown.R#L95-L106), which absolutely cannot work if the Pandoc tool is unavailable:
Copy file name to clipboardExpand all lines: vignettes/snapshotting.Rmd
+4-4Lines changed: 4 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -16,16 +16,16 @@ set.seed(1014)
16
16
```
17
17
18
18
The goal of a unit test is to record the expected output of a function using code.
19
-
This is a powerful technique because not only does it ensure that code doesn't change unexpectedly, it also expresses the desired behaviour in a way that a human can understand.
19
+
This is a powerful technique because not only does it ensure that code doesn't change unexpectedly, but it also expresses the desired behavior in a way that a human can understand.
20
20
21
-
However, it's not always convenient to record the expected behaviour with code.
21
+
However, it's not always convenient to record the expected behavior with code.
22
22
Some challenges include:
23
23
24
24
- Text output that includes many characters like quotes and newlines that require special handling in a string.
25
25
26
26
- Output that is large, making it painful to define the reference output and bloating the size of the test file.
27
27
28
-
- Binary formats like plots or images, which are very difficult to describe in code: e.g. the plot looks right, the error message is actionable, or the print method uses colour effectively.
28
+
- Binary formats like plots or images, which are very difficult to describe in code: e.g., the plot looks right, the error message is actionable, or the print method uses colour effectively.
29
29
30
30
For these situations, testthat provides an alternative mechanism: snapshot tests.
31
31
Instead of using code to describe expected output, snapshot tests (also known as [golden tests](https://ro-che.info/articles/2017-12-04-golden-tests)) record results in a separate human readable file.
When we run the test for the first time, it automatically generates reference output, and prints it, so that you can visually confirm that it's correct.
93
93
The output is automatically saved in `_snaps/{name}.md`.
94
-
The name of the snapshot matches your test file name --- e.g. if your test is `test-pizza.R` then your snapshot will be saved in `test/testthat/_snaps/pizza.md`.
94
+
The name of the snapshot matches your test file name --- e.g. if your test is `test-pizza.R` then your snapshot will be saved in `tests/testthat/_snaps/pizza.md`.
95
95
As the file name suggests, this is a markdown file, which I'll explain shortly.
Copy file name to clipboardExpand all lines: vignettes/test-fixtures.Rmd
+6-6Lines changed: 6 additions & 6 deletions
Original file line number
Diff line number
Diff line change
@@ -20,7 +20,7 @@ knitr::opts_chunk$set(
20
20
>
21
21
> ― Chief Si'ahl
22
22
23
-
Ideally, a test should leave the world exactly as it found it. But you often need to make some changes in order to exercise every part of your code:
23
+
Ideally, a test should leave the world exactly as it found it. But you often need to make some changes to exercise every part of your code:
24
24
25
25
- Create a file or directory
26
26
- Create a resource on an external system
@@ -31,9 +31,9 @@ Ideally, a test should leave the world exactly as it found it. But you often nee
31
31
32
32
How can you clean up these changes to get back to a clean slate? Scrupulous attention to cleanup is more than just courtesy or being fastidious. It is also self-serving. The state of the world after test `i` is the starting state for test `i + 1`. Tests that change state willy-nilly eventually end up interfering with each other in ways that can be very difficult to debug.
33
33
34
-
Most tests are written with an implicit assumption about the starting state, usually whatever *tabula rasa* means for the target domain of your package. If you accumulate enough sloppy tests, you will eventually find yourself asking the programming equivalent of questions like "Who forgot to turn off the oven?" and "Who didn't clean up after the dog?". (If you've got yourself into this state, testthat provides another tool to help you figure out exactly what test is to blame: `set_state_inspector()`.)
34
+
Most tests are written with an implicit assumption about the starting state, usually whatever *tabula rasa* means for the target domain of your package. If you accumulate enough sloppy tests, you will eventually find yourself asking the programming equivalent of questions like "Who forgot to turn off the oven?" and "Who didn't clean up after the dog?" (If you've got yourself into this state, testthat provides another tool to help you figure out exactly what test is to blame: `set_state_inspector()`.)
35
35
36
-
It's also important that your setup and cleanup is easy to use when working interactively. When a test fails, you want to be able to quickly recreate the exact environment in which the test is run so you can interactively experiment to figure out what went wrong.
36
+
It's also important that your setup and cleanup are easy to use when working interactively. When a test fails, you want to be able to quickly recreate the exact environment in which the test is run so you can interactively experiment to figure out what went wrong.
37
37
38
38
This article introduces a powerful technique that allows you to solve both problems: **test fixtures**. We'll begin by discussing some pre-existing tools, then learn about the tools that make fixtures possible, talk about exactly what a test fixture is, and show a few examples.
39
39
@@ -45,7 +45,7 @@ library(testthat)
45
45
46
46
## `local_` helpers
47
47
48
-
We'll begin by giving you the bare minimum of knowledge to change global state _just_ within your test. The withr package provides a number of functions that temporarily change the state of the world, carefully undoing the changes when the current function finishes:
48
+
We'll begin by giving you the bare minimum of knowledge to change global state *just* within your test. The withr package provides a number of functions that temporarily change the state of the world, carefully undoing the changes when the current function finishes:
@@ -57,7 +57,7 @@ We'll begin by giving you the bare minimum of knowledge to change global state _
57
57
58
58
(You can see a full list at <https://withr.r-lib.org/> but these five are by far the most commonly used.)
59
59
60
-
These allow you to control options that would otherwise be painful. For example, imagine you were testing the base R code that rounds numbers to a fixed number of places when printing. You could write code like this:
60
+
These allow you to control options that would otherwise be painful. For example, imagine you were testing base R code that rounds numbers to a fixed number of places when printing. You could write code like this:
This code doesn't work because the cleanup happens too soon, when `local_digits()` exits, not when `neat()` finishes.
205
+
This code doesn't work because the cleanup happens too soon, when `local_digits()` exits, not when `neater()` finishes.
206
206
207
207
Fortunately, `withr::defer()` allows us to solve this problem by providing an `envir` argument that allows you to control when cleanup occurs. The exact details of how this works are rather complicated, but fortunately there's a common pattern you can use without understanding all the details. Your helper function should always have an `env` argument that defaults to `parent.frame()`, which you pass to the second argument of `defer()`:
0 commit comments