should `test_check(stop_on_warning = TRUE)` throw an error for warning about "code run outside of test_that()"?

[d-morrison]

I recently encountered a test file with a structure like this:

test_that(
  desc = "no error",
  code = expect_no_error(
    {1 + 1}
  )
)

When I test this file, it warns about code run outside of test_that(), as expected:

> devtools:::test_active_file()
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 6 ]

── Warning (test-autoplot.summary.seroincidence.by.R:108:1): (code run outside of `test_that()`) ──────
The `code` argument to `test_that()` must be a braced expression to get accurate file-line information for failures.
Backtrace:
    ▆
 1. └─testthat::test_that(...) at test-autoplot.summary.seroincidence.by.R:108:1

── Warning (test-autoplot.summary.seroincidence.by.R:145:1): (code run outside of `test_that()`) ──────
The `code` argument to `test_that()` must be a braced expression to get accurate file-line information for failures.
Backtrace:
    ▆
 1. └─testthat::test_that(...) at test-autoplot.summary.seroincidence.by.R:145:1
[ FAIL 0 | WARN 2 | SKIP 0 | PASS 6 ]

However, when I run devtools::check(error_on = "warning") on the package containing this test, devtools::check() doesn't seem to count the testthat warning as real, even when I set test_check("[packagename]", stop_on_warning = TRUE) in testthat.R:

── R CMD check results ───────────────────────────────────────────────── 
Duration: 2m 20.3s

0 errors ✔ | 0 warnings ✔ | 0 notes ✔

I wrote a new test in a fork of testthat as a reprex: #2069

Is this behavior as-intended?

[hadley]

Yes, that's expected. Warnings from testthat deliberately do not become warnings in R CMD check. Otherwise (e.g.) deprecation warnings would cause R CMD check to fail on CRAN, thus defeating the purpose of deprecation.