|
| 1 | +--- |
| 2 | +title: "Testing challenging functions" |
| 3 | +output: rmarkdown::html_vignette |
| 4 | +vignette: > |
| 5 | + %\VignetteIndexEntry{Testing challenging functions} |
| 6 | + %\VignetteEngine{knitr::rmarkdown} |
| 7 | + %\VignetteEncoding{UTF-8} |
| 8 | +--- |
| 9 | + |
| 10 | +```{r} |
| 11 | +#| include: false |
| 12 | +library(testthat) |
| 13 | +knitr::opts_chunk$set(collapse = TRUE, comment = "#>") |
| 14 | +
|
| 15 | +# Pretend we're snapshotting |
| 16 | +snapper <- local_snapshotter(fail_on_new = FALSE) |
| 17 | +snapper$start_file("snapshotting.Rmd", "test") |
| 18 | +
|
| 19 | +# Pretend we're testing testthat so we can use mocking |
| 20 | +Sys.setenv(TESTTHAT_PKG = "testthat") |
| 21 | +``` |
| 22 | + |
| 23 | +This vignette is a quick reference guide for testing challenging functions. It's organized by problem type rather than technique, so you can quickly skim the whole vignette, spot the problem you're facing, and then learn more about useful tools for solving it. In it, you'll learn how to overcome the following challenges: |
| 24 | + |
| 25 | +* Functions with implicit inputs, like options and environment variables. |
| 26 | +* Random number generators. |
| 27 | +* Tests that can't be run in some environments. |
| 28 | +* Testing web APIs. |
| 29 | +* Testing graphical output. |
| 30 | +* User interaction. |
| 31 | +* User-facing text. |
| 32 | +* Repeated code. |
| 33 | + |
| 34 | +## Options and environment variables |
| 35 | + |
| 36 | +If your function depends on options or environment variables, first try refactoring the function to make the [inputs explicit](https://design.tidyverse.org/inputs-explicit.html). If that's not possible, use functions like `withr::local_options()` or `withr::local_envvar()` to temporarily change options and environment values within a test. Learn more in `vignette("test-fixtures")`. |
| 37 | + |
| 38 | +<!-- FIXME: Consider adding a brief example showing the difference between implicit and explicit approaches - this would make the recommendation more concrete --> |
| 39 | + |
| 40 | +## Random numbers |
| 41 | + |
| 42 | +What happens if you want to test a function that relies on randomness in some way? If you're writing a random number generator, you probably want to generate a large quantity of random numbers and then apply some statistical test. But what if your function just happens to use a little bit of pre-existing randomness? How do you make your tests repeatable and reproducible? Under the hood, random number generators generate different numbers because they update a special `.Random.seed` variable stored in the global environment. You can temporarily set this seed to a known value to make your random numbers reproducible with `withr::local_seed()`, making random numbers a special case of test fixtures (`vignette("test-fixtures")`). |
| 43 | + |
| 44 | +Here's a simple example showing how you might test the basic operation of a function that rolls a die: |
| 45 | + |
| 46 | +```{r} |
| 47 | +#| label: random-local-seed |
| 48 | +dice <- function() { |
| 49 | + sample(6, 1) |
| 50 | +} |
| 51 | +
|
| 52 | +test_that("dice returns different numbers", { |
| 53 | + withr::local_seed(1234) |
| 54 | +
|
| 55 | + expect_equal(dice(), 4) |
| 56 | + expect_equal(dice(), 2) |
| 57 | + expect_equal(dice(), 6) |
| 58 | +}) |
| 59 | +``` |
| 60 | + |
| 61 | +Alternatively, you might want to mock (`vignette("mocking")`) the function to eliminate randomness. |
| 62 | + |
| 63 | +```{r} |
| 64 | +#| label: random-mock |
| 65 | +
|
| 66 | +roll_three <- function() { |
| 67 | + sum(dice(), dice(), dice()) |
| 68 | +} |
| 69 | +
|
| 70 | +test_that("three dice adds values of individual calls", { |
| 71 | + local_mocked_bindings(dice = mock_output_sequence(1, 2, 3)) |
| 72 | + expect_equal(roll_three(), 6) |
| 73 | +}) |
| 74 | +``` |
| 75 | + |
| 76 | +When should you set the seed and when should you use mocking? As a general rule of thumb, set the seed when you want to test the actual random behavior, and use mocking when you want to test the logic that uses the random results. |
| 77 | + |
| 78 | +## Some tests can't be run in some circumstances |
| 79 | + |
| 80 | +You can skip a test without it passing or failing if you can't or don't want to run it (e.g., it's OS dependent, it only works interactively, or it shouldn't be tested on CRAN). Learn more in `vignette("skipping")`. |
| 81 | + |
| 82 | +## HTTP requests |
| 83 | + |
| 84 | +If you're trying to test functions that rely on HTTP requests, we recommend using {vcr} or {httptest2}. These packages both allow you to interactively record HTTP responses and then later replay them in tests. This is a specialized type of mocking (`vignette("mocking")`) that works with {httr} and {httr2} to isolates your tests from failures in the underlying API. |
| 85 | + |
| 86 | +If your package is going to CRAN, you **must** either use one of these packages or use `skip_on_cran()` for all internet-facing tests. Otherwise, you are at high risk of failing `R CMD check` if the underlying API is temporarily down. This sort of failure causes extra work for the CRAN maintainers and extra hassle for you. |
| 87 | + |
| 88 | +## Graphics |
| 89 | + |
| 90 | +The only type of testing you can use for graphics is snapshot testing (`vignette("snapshotting")`) via `expect_snapshot_file()`. Graphical snapshot testing is surprisingly challenging because you need pixel-perfect rendering across multiple versions of multiple operating systems, and this is hard, mostly due to imperceptble differences in font rendering. Fortunately we've needed to overcome these challenges in order to test ggplot2, and you can benefit from our experience by using {vdiffr} when testing graphical output. |
| 91 | + |
| 92 | +## User interaction |
| 93 | + |
| 94 | +If you're testing a function that relies on user feedback (e.g. from `readline()`, `utils::menu()`, or `utils::askYesNo()`), you can use mocking (`vignette("mocking")`) to return fixed values within the test. For example, imagine that you've written the following function that asks the user if they want to continue: |
| 95 | + |
| 96 | +```{r} |
| 97 | +#| label: continue |
| 98 | +
|
| 99 | +continue <- function(prompt) { |
| 100 | + cat(prompt, "\n", sep = "") |
| 101 | +
|
| 102 | + repeat { |
| 103 | + val <- readline("Do you want to continue? (y/n) ") |
| 104 | + if (val %in% c("y", "n")) { |
| 105 | + return(val == "y") |
| 106 | + } |
| 107 | + cat("! You must enter y or n\n") |
| 108 | + } |
| 109 | +} |
| 110 | +
|
| 111 | +readline <- NULL |
| 112 | +``` |
| 113 | + |
| 114 | +You could test its behavior by mocking `readline()` and using a snapshot test: |
| 115 | + |
| 116 | +```{r} |
| 117 | +#| label: mock-readline |
| 118 | +
|
| 119 | +test_that("user must respond y or n", { |
| 120 | + mock_readline <- local({ |
| 121 | + i <- 0 |
| 122 | + function(prompt) { |
| 123 | + i <<- i + 1 |
| 124 | + cat(prompt) |
| 125 | + val <- if (i == 1) "x" else "y" |
| 126 | + cat(val, "\n", sep = "") |
| 127 | + val |
| 128 | + } |
| 129 | + }) |
| 130 | +
|
| 131 | + local_mocked_bindings(readline = mock_readline) |
| 132 | + expect_snapshot(val <- continue("This is dangerous")) |
| 133 | + expect_true(val) |
| 134 | +}) |
| 135 | +``` |
| 136 | + |
| 137 | +If you were testing the behavior of some function that uses `continue()`, you might want to mock `continue()` instead of `readline()`. For example, the function below requires user confirmation before overwriting an existing file. In order to focus our tests on the behavior of just this function, we mock `continue()` to return either `TRUE` or `FALSE` without any user messaging. |
| 138 | + |
| 139 | +```{r} |
| 140 | +#| label: mock-continue |
| 141 | +
|
| 142 | +save_file <- function(path, data) { |
| 143 | + if (file.exists(path)) { |
| 144 | + if (!continue("`path` already exists")) { |
| 145 | + stop("Failed to continue") |
| 146 | + } |
| 147 | + } |
| 148 | + writeLines(data, path) |
| 149 | +} |
| 150 | +
|
| 151 | +test_that("save_file() requires confirmation to overwrite file", { |
| 152 | + path <- withr::local_tempfile(lines = letters) |
| 153 | +
|
| 154 | + local_mocked_bindings(continue = function(...) TRUE) |
| 155 | + save_file(path, "a") |
| 156 | + expect_equal(readLines(path), "a") |
| 157 | +
|
| 158 | + local_mocked_bindings(continue = function(...) FALSE) |
| 159 | + expect_snapshot(save_file(path, "a"), error = TRUE) |
| 160 | +}) |
| 161 | +``` |
| 162 | + |
| 163 | +## User-facing text |
| 164 | + |
| 165 | +Errors, warnings, and other user-facing text should be tested to ensure they're both actionable and consistent across the package. Obviously, it's not possible to test this automatically, but you can use snapshots (`vignette("snapshotting")`) to ensure that user-facing messages are clearly shown in PRs and easily reviewed by another human. |
| 166 | + |
| 167 | +## Repeated code |
| 168 | + |
| 169 | +If you find yourself repeating the same set of expectations again and again across your test suite, it may be a sign that you should design your own expectation. Learn how in `vignette("custom-expectations")`. |
0 commit comments