How to check both output and an error?

[MichaelChirico]

I have some code that works like this:

check <- function(...) {
  writeLines(c(...))
  stop(...)
}

The idea is that because the stop() buffer is pretty limited, I opt to dump a potentially large amount of information with writeLines(), then follow up by signalling an error like "Failed; see above".

I expected {testthat} code like the following to work:

expect_error(
  expect_output(
    check(...),
    "outputMatch"
  ),
  "errorMatch"
)

And I thought all was well, however, now I'm noticing this isn't working like I'd intended at all. Basically, I'm not sure if expect_output() is being run at all. Observe:

expect_error(
  expect_output(
    { cat("foo\n"); stop("bar") },
    "xxx" # not a match
  ),
  "bar" # a match
)
# passes, no error from `expect_output()` failing!

Somewhat more insidiously, we don't even get notified if we've totally botched the input to expect_output():

expect_error(
  expect_output(
    { cat("foo\n"); stop("bar") },
    oopsidaisy() # not defined!
  ),
  "bar" # a match
)
# passes, no error from `expect_output()` failing!

(In some sense, that's the same issue as above, since abstractly both are errors from the expect_output() expression, albeit with different condition classes)

[MichaelChirico]

Apparently reversing the order is what's needed

expect_output(
  expect_error(
    { cat("foo\n"); stop("bar") },
    "bar" # a match
  ),
  "xxx" # not a match
)
# Error: `expect_error\(\.\.\.\)` does not match "xxx".
# Actual value: "foo"

Is that working as intended? I wrote it as:

code |>
  expect_output() |>
  expect_error()

Because that's the logical ordering that matches the order the signals are produced in the code

---

Aside: the expect_output() failure here could go for improvement -- it's not clear in this nested scenario that "Error:" and "Actual value:" are associated with a failure of output to match.

[hadley]

Hmmmm, I think this reveals the core issue:

library(testthat)

test_that("", {
  expect_output(
    {
      cat("foo\n")
      stop("bar")
    },
    oopsidaisy() # not defined!
  )
})
#> ── Error:  ─────────────────────────────────────────────────────────────────────
#> Error in `eval_bare(quo_get_expr(.quo), quo_get_env(.quo))`: bar
#> Backtrace:
#>     ▆
#>  1. └─testthat::expect_output(...)
#>  2.   └─testthat:::quasi_capture(...)
#>  3.     ├─testthat (local) .capture(...)
#>  4.     │ └─testthat::capture_output_lines(code, print, width = width)
#>  5.     │   └─testthat:::eval_with_output(code, print = print, width = width)
#>  6.     │     ├─withr::with_output_sink(path, withVisible(code))
#>  7.     │     │ └─base::force(code)
#>  8.     │     └─base::withVisible(code)
#>  9.     └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
#> Error:
#> ! Test failed with 1 failure and 0 successes.

^{Created on 2025-10-24 with reprex v2.1.1}

i.e. if code instead an expectation errors, the expectation does not fail. And then you catch that error with expect_error() so the whole thing looks like it's ok. I'm not sure there's anything we can do here since it all seems like reasonable applications of basic principles.

You are slightly more likely to catch the problem with dev testthat since it now reports the number of successes and failures:

library(testthat)
test_that("", {
  expect_error(
    expect_output(
      {
        cat("foo\n")
        stop("bar")
      },
      oopsidaisy() # not defined!
    ),
    "bar" # a match
  )
})
#> Test passed with 1 success 🌈.

^{Created on 2025-10-24 with reprex v2.1.1}

[MichaelChirico]

I guess the {testthat}-independent way to look at this is as follows:

capture.output(tryCatch({ cat("foo\n"); stop("bar") }, error=identity))
# [1] "foo"                                                                      "<simpleError in doTryCatch(return(expr), name, parentenv, handler): bar>"
tryCatch(capture.output({ cat("foo\n"); stop("bar") }), error=identity)
# <simpleError in withVisible(...elt(i)): bar>

i.e., we have to catch the error in the expression whose output we wish to capture, if we actually wish to see the output as well.

Is this worth maybe adding as an example in ?expect_output?