Add new special files vignette (#1769)

Fixes #1638

Author: hadley

NEWS.md

@@ -1,5 +1,8 @@
 # testthat (development version)
 
+* New `vignette("special-files")` describes the various special files
+  that testthat uses (#1638).
+
 * `skip_on_bioc()` now uses the documented environment variable
   (`IS_BIOC_BUILD_MACHINE`) (#1712).
 

R/test-files.R

@@ -1,35 +1,13 @@
 #' Run all tests in a directory
 #'
+#' @description
 #' This function is the low-level workhorse that powers [test_local()] and
 #' [test_package()]. Generally, you should not call this function directly.
 #' In particular, you are responsible for ensuring that the functions to test
 #' are available in the test `env` (e.g. via `load_package`).
 #'
-#' @section Special files:
-#' Certain `.R` files have special significance in testthat:
-#'
-#' * Test files start with `test` and are executed in alphabetical order.
-#'
-#' * Setup files start with `setup` and are executed before tests. If
-#'   clean up is needed after all tests have been run, you can use
-#'   `withr::defer(clean_up(), teardown_env())`. See `vignette("test-fixtures")`
-#'   for more details.
-#'
-#' * Helper files start with `helper` and are executed before tests are
-#'   run and, unlike setup files, are also loaded by `devtools::load_all()`.
-#'   Helper files can be necessary for side-effect-y code that you need to run
-#'   when developing the package interactively. It's certainly possible to
-#'   define custom test utilities in a helper file, but they can usually be
-#'   defined below `R/`, just like any other internal function.
-#'
-#' There is another type of special file that we no longer recommend using:
-#'
-#' * Teardown files start with `teardown` and are executed after the tests
-#'   are run. Now we recommend interleaving setup and cleanup code in `setup-`
-#'   files, making it easier to check that you automatically clean up every
-#'   mess that you make.
-#'
-#' All other files are ignored by testthat.
+#' See `vignette("special-files")` to learn more about the conventions for test,
+#' helper, and setup files that testthat uses, and what you might use each for.
 #'
 #' @section Environments:
 #' Each test is run in a clean environment to keep tests as isolated as
@@ -45,7 +23,6 @@
 #' @param env Environment in which to execute the tests. Expert use only.
 #' @param ... Additional arguments passed to [grepl()] to control filtering.
 #' @param load_helpers Source helper files before running the tests?
-#'   See [source_test_helpers()] for more details.
 #' @param stop_on_failure If `TRUE`, throw an error if any tests fail.
 #' @param stop_on_warning If `TRUE`, throw an error if any tests generate
 #'   warnings.
@@ -126,10 +103,9 @@ test_dir <- function(path,
 #' Run all tests in a single file
 #'
 #' Helper, setup, and teardown files located in the same directory as the
-#' test will also be run.
+#' test will also be run. See `vignette("special-files")` for details.
 #'
 #' @inherit test_dir return params
-#' @inheritSection test_dir Special files
 #' @inheritSection test_dir Environments
 #' @param path Path to file.
 #' @param ... Additional parameters passed on to `test_dir()`

R/test-package.R

@@ -5,7 +5,8 @@
 #' * `test_package()` tests an installed package.
 #' * `test_check()` checks a package during `R CMD check`.
 #'
-#' Tests live in `tests/testthat`.
+#' See `vignette("special-files")` to learn about the various files that
+#' testthat works with.
 #'
 #' @section `R CMD check`:
 #' To run testthat automatically from `R CMD check`, make sure you have
@@ -19,7 +20,6 @@
 #' ```
 #'
 #' @inherit test_dir return params
-#' @inheritSection test_dir Special files
 #' @inheritSection test_dir Environments
 #' @param ... Additional arguments passed to [test_dir()]
 #' @export

man/test_dir.Rd

@@ -12,9 +12,9 @@ library(testthat)
 knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
 ```
 
-This vignette shows you how to create custom expectations that work identically to the built-in `expect_` functions.  Since these functions will need to be loaded when your package is loaded for testing, it is recommended that `expect_` functions be defined in `test-helpers.R` in your packages `R/` directory.
+This vignette shows you how to create custom expectations that work identically to the built-in `expect_` functions. Since these functions will need to be loaded in order for your tests to work, we recommend putting them in an appropriately named helper file, i.e. `tests/testthat/helper-expectations.R`.
 
-## Creating an expectation
+## Expectation basics
 
 There are three main parts to writing an expectation, as illustrated by `expect_length()`:
 
@@ -35,7 +35,7 @@ expect_length <- function(object, n) {
 }
 ```
 
-## Quasi-labelling
+### Quasi-labelling
 
 The first step in any expectation is to capture the actual object, and generate a label for it to use if a failure occur. All testthat expectations support quasiquotation so that you can unquote variables. This makes it easier to generate good labels when the expectation is called from a function or within a for loop.
 
@@ -86,16 +86,3 @@ expect_length <- function(object, n) {
   fail(message)
 }
 ```
-
-## Testing your expectations
-
-Use the expectations `expect_success()` and `expect_failure()` to test your expectation.
-
-```{r}
-test_that("length computed correctly", {
-  expect_success(expect_length(1, 1))
-  expect_failure(expect_length(1, 2), "has length 1, not length 2.")
-  expect_success(expect_length(1:10, 10))
-  expect_success(expect_length(letters[1:5], 5))
-})
-```

man/test_file.Rd

@@ -0,0 +1,88 @@
+---
+title: "Special files"
+output: rmarkdown::html_vignette
+vignette: >
+  %\VignetteIndexEntry{Special files}
+  %\VignetteEngine{knitr::rmarkdown}
+  %\VignetteEncoding{UTF-8}
+---
+
+```{r, include = FALSE}
+knitr::opts_chunk$set(
+  collapse = TRUE,
+  comment = "#>"
+)
+```
+
+```{r setup}
+#| include: false
+library(testthat)
+```
+
+This vignette describes the various special files that testthat understands: test, helper, setup/teardown, snapshot, and everything else.
+
+## Test files
+
+These are the bread and butter of testthat.
+Test files live in `tests/testthat/`, start with either `test-` or `test_`, and end with `.r` or `.R`.
+We recommend organising your test files so that there's a one-to-one correspondence between the files in `R/` and the files in `tests/testthat/` so that (e.g.) `R/myfile.R` has a matching `tests/testthat/test-myfile.R`.
+This correspondence is maintained by functions like `usethis::use_r()` and `usethis::use_test()` and is taken advantage of by functions like `devtools::test_active_file()` and `devtools::test_coverage_active_file()`.
+
+Test files are executed in alphabetical order, but you should strive to avoid dependencies between test files.
+In principle, you should be able to be run your test files in any order or even at the same time.
+
+## Helper files
+
+Helper files live in `tests/testtthat/`, start with `helper`, and end with `.r` or `.R`.
+They are sourced by `devtools::load_all()` (so they're available interactively when developing your packages) and by `test_check()` and friends (so that they're available no matter how your tests are executed).
+
+Helper files are a useful place for functions that you've extracted from repeated code in your tests, whether that be test fixtures (`vignette("test-fixtures")`), custom expectations (`vignette("custom-expectations")`), or skip helpers (`vignette("skipping")`).
+
+## Setup files
+
+Setup files live in `tests/testthat/`, start with `setup`, and end with `.r` or `.R`.
+Typically there is only one setup file which, by convention, is `tests/testthat/setup.R`.
+Setup files are sourced by `test_check()` and friends (so that they're available no matter how your tests are executed), but they are *not* sourced by `devtools::load_all()`.
+
+Setup files are good place to put truly global test setup that would be impractical to build into every single test and that might be tailored for test execution in non-interactive or remote environments.
+Examples:
+
+-   Turning off behaviour aimed at an interactive user, such as messaging or writing to the clipboard.
+-   Setting up a cache folder.
+
+If any of your setup should be reversed after test execution (i.e. it needs to be torn down), we recommend maintaining that teardown code alongside the setup code, in `setup.R`, because this makes it easier to ensure they stay in sync.
+The artificial environment `teardown_env()` exists as a magical handle to use in `withr::defer()` and `withr::local_*()`.
+A legacy approach (which still works, but is no longer recommended) is to put teardown code in `tests/testthat/teardown.R`.
+
+Here's a `setup.R` example from the reprex package, where we turn off clipboard and HTML preview functionality during testing:
+
+```{r eval = FALSE}
+op <- options(reprex.clipboard = FALSE, reprex.html_preview = FALSE)
+
+withr::defer(options(op), teardown_env())
+```
+
+Since we are just modifying options here, we can be even more concise and use the pre-built function `withr::local_options()` and pass `teardown_env()` as the `.local_envir`:
+
+```{r eval = FALSE}
+withr::local_options(
+  list(reprex.clipboard = FALSE, reprex.html_preview = FALSE),
+  .local_envir = teardown_env()
+)
+```
+
+### Teardown files
+
+Teardown files live in `tests/testhat/`, start with `teardown` and end with `.r` or `.R`.
+They are executed after the tests are run, but we no longer recommend using them as it's easier to check that you clean up every mess that you make if you interleave setup and tear down code as described above.
+
+## Snapshot files
+
+Snapshot files live in `tests/testthat/_snaps/`.
+Snapshot file are named automatically based on the name of the test file so that `tests/testthat/test-one.R` will generated `tests/testthat/_snaps/one.md`.
+Learn more about snapshot tests in `vignette("snapshotting")`.
+
+## Other files and folders
+
+Other files and folders in `tests/testthat/` are ignored by testthat, making them a good place to store persistent test data.
+Since the precise location of the `test/testthat/` directory varies slightly depending on how you're running the test, we recommend creating paths to these files and directories using `test_path()`.