Skip to content

Commit 40573f0

Browse files
hadleyhadley
committed
✨ Respect max_failures in ParallelProgressReporter ✨
This is so surprisingly simple that I wonder if I (or more corretly Claude) has missed something. I also made the somewhat related tweak to not do the final report (which repeats all failures etc) in the case of an early exit. Fixes #1162
1 parent e5cd770 commit 40573f0

File tree

5 files changed

+113
-82
lines changed

5 files changed

+113
-82
lines changed

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ General advice:
1616

1717
- `devtools::test()` - Run all tests
1818
- `devtools::test_file("tests/testthat/test-filename.R")` - Run tests in a specific file
19+
- DO NOT USE `devtools::test_active_file()`
1920
- `devtools::load_all()` - Load package for development
2021
- `devtools::document()` - Generate documentation
2122
- `devtools::check()` - Run R CMD check

‎NEWS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
# testthat (development version)
22

3+
* `ParallelProgressReporter` now respect `max_failures` (#1162).
34
* `JunitReporter()` no longer fails with `"no applicable method for xml_add_child"` for warnings outside of tests (#1913). Additionally, warnings now save their backtraces.
45
* `JunitReporter()` strips ANSI escapes in more placese (#1852, #2032).
56
* `try_again()` is now publicised. The first argument is now the number of retries, not tries (#2050).

‎R/reporter-progress.R‎

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -221,16 +221,20 @@ ProgressReporter <- R6::R6Class(
221221
self$report_issues(self$ctxt_issues)
222222

223223
if (self$is_full()) {
224-
snapshotter <- get_snapshotter()
225-
if (!is.null(snapshotter)) {
226-
snapshotter$end_file()
227-
}
224+
self$report_full()
225+
}
226+
},
228227

229-
stop_reporter(c(
230-
"Maximum number of failures exceeded; quitting at end of file.",
231-
i = "Increase this number with (e.g.) {.run testthat::set_max_fails(Inf)}"
232-
))
228+
report_full = function() {
229+
snapshotter <- get_snapshotter()
230+
if (!is.null(snapshotter)) {
231+
snapshotter$end_file()
233232
}
233+
234+
stop_reporter(c(
235+
"Maximum number of failures exceeded; quitting at end of file.",
236+
i = "Increase this number with (e.g.) {.run testthat::set_max_fails(Inf)}"
237+
))
234238
},
235239

236240
add_result = function(context, test, result) {
@@ -258,6 +262,10 @@ ProgressReporter <- R6::R6Class(
258262
},
259263

260264
end_reporter = function() {
265+
if (self$is_full()) {
266+
return()
267+
}
268+
261269
self$cat_line()
262270

263271
colour_if <- function(n, type) {
@@ -465,7 +473,13 @@ ParallelProgressReporter <- R6::R6Class(
465473
self$report_issues(fsts$issues)
466474

467475
self$files[[self$file_name]] <- NULL
468-
if (length(self$files)) self$update(force = TRUE)
476+
if (length(self$files)) {
477+
self$update(force = TRUE)
478+
}
479+
480+
if (self$is_full()) {
481+
self$report_full()
482+
}
469483
},
470484

471485
end_reporter = function() {

‎tests/testthat/_snaps/reporter-progress.md‎

Lines changed: 78 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -133,79 +133,6 @@
133133
--------------------------------------------------------------------------------
134134
Maximum number of failures exceeded; quitting at end of file.
135135
i Increase this number with (e.g.) `testthat::set_max_fails(Inf)`
136-
137-
== Results =====================================================================
138-
-- Failed tests ----------------------------------------------------------------
139-
Failure ('reporters/fail-many.R:3:5'): Example
140-
FALSE (`actual`) is not equal to TRUE (`expected`).
141-
142-
`actual`: FALSE
143-
`expected`: TRUE
144-
145-
Failure ('reporters/fail-many.R:3:5'): Example
146-
FALSE (`actual`) is not equal to TRUE (`expected`).
147-
148-
`actual`: FALSE
149-
`expected`: TRUE
150-
151-
Failure ('reporters/fail-many.R:3:5'): Example
152-
FALSE (`actual`) is not equal to TRUE (`expected`).
153-
154-
`actual`: FALSE
155-
`expected`: TRUE
156-
157-
Failure ('reporters/fail-many.R:3:5'): Example
158-
FALSE (`actual`) is not equal to TRUE (`expected`).
159-
160-
`actual`: FALSE
161-
`expected`: TRUE
162-
163-
Failure ('reporters/fail-many.R:3:5'): Example
164-
FALSE (`actual`) is not equal to TRUE (`expected`).
165-
166-
`actual`: FALSE
167-
`expected`: TRUE
168-
169-
Failure ('reporters/fail-many.R:3:5'): Example
170-
FALSE (`actual`) is not equal to TRUE (`expected`).
171-
172-
`actual`: FALSE
173-
`expected`: TRUE
174-
175-
Failure ('reporters/fail-many.R:3:5'): Example
176-
FALSE (`actual`) is not equal to TRUE (`expected`).
177-
178-
`actual`: FALSE
179-
`expected`: TRUE
180-
181-
Failure ('reporters/fail-many.R:3:5'): Example
182-
FALSE (`actual`) is not equal to TRUE (`expected`).
183-
184-
`actual`: FALSE
185-
`expected`: TRUE
186-
187-
Failure ('reporters/fail-many.R:3:5'): Example
188-
FALSE (`actual`) is not equal to TRUE (`expected`).
189-
190-
`actual`: FALSE
191-
`expected`: TRUE
192-
193-
Failure ('reporters/fail-many.R:3:5'): Example
194-
FALSE (`actual`) is not equal to TRUE (`expected`).
195-
196-
`actual`: FALSE
197-
`expected`: TRUE
198-
199-
Failure ('reporters/fail-many.R:3:5'): Example
200-
FALSE (`actual`) is not equal to TRUE (`expected`).
201-
202-
`actual`: FALSE
203-
`expected`: TRUE
204-
205-
[ FAIL 11 | WARN 0 | SKIP 0 | PASS 0 ]
206-
== Terminated early ============================================================
207-
208-
No one gets it right on their first try
209136

210137
# can fully suppress incremental updates
211138

@@ -632,3 +559,81 @@
632559
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 6 ]
633560
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 7 ]
634561

562+
# ParallelProgressReporter fails after max_fail tests
563+
564+
v | F W S OK | Context
565+
566+
- [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...
567+
x | 11 0 | reporters/fail-many
568+
--------------------------------------------------------------------------------
569+
Failure ('reporters/fail-many.R:3:5'): Example
570+
FALSE (`actual`) is not equal to TRUE (`expected`).
571+
572+
`actual`: FALSE
573+
`expected`: TRUE
574+
575+
Failure ('reporters/fail-many.R:3:5'): Example
576+
FALSE (`actual`) is not equal to TRUE (`expected`).
577+
578+
`actual`: FALSE
579+
`expected`: TRUE
580+
581+
Failure ('reporters/fail-many.R:3:5'): Example
582+
FALSE (`actual`) is not equal to TRUE (`expected`).
583+
584+
`actual`: FALSE
585+
`expected`: TRUE
586+
587+
Failure ('reporters/fail-many.R:3:5'): Example
588+
FALSE (`actual`) is not equal to TRUE (`expected`).
589+
590+
`actual`: FALSE
591+
`expected`: TRUE
592+
593+
Failure ('reporters/fail-many.R:3:5'): Example
594+
FALSE (`actual`) is not equal to TRUE (`expected`).
595+
596+
`actual`: FALSE
597+
`expected`: TRUE
598+
599+
Failure ('reporters/fail-many.R:3:5'): Example
600+
FALSE (`actual`) is not equal to TRUE (`expected`).
601+
602+
`actual`: FALSE
603+
`expected`: TRUE
604+
605+
Failure ('reporters/fail-many.R:3:5'): Example
606+
FALSE (`actual`) is not equal to TRUE (`expected`).
607+
608+
`actual`: FALSE
609+
`expected`: TRUE
610+
611+
Failure ('reporters/fail-many.R:3:5'): Example
612+
FALSE (`actual`) is not equal to TRUE (`expected`).
613+
614+
`actual`: FALSE
615+
`expected`: TRUE
616+
617+
Failure ('reporters/fail-many.R:3:5'): Example
618+
FALSE (`actual`) is not equal to TRUE (`expected`).
619+
620+
`actual`: FALSE
621+
`expected`: TRUE
622+
623+
Failure ('reporters/fail-many.R:3:5'): Example
624+
FALSE (`actual`) is not equal to TRUE (`expected`).
625+
626+
`actual`: FALSE
627+
`expected`: TRUE
628+
629+
Failure ('reporters/fail-many.R:3:5'): Example
630+
FALSE (`actual`) is not equal to TRUE (`expected`).
631+
632+
`actual`: FALSE
633+
`expected`: TRUE
634+
--------------------------------------------------------------------------------
635+
Maximum number of failures exceeded; quitting at end of file.
636+
i Increase this number with (e.g.) `testthat::set_max_fails(Inf)`
637+
638+
639+

‎tests/testthat/test-reporter-progress.R‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,3 +83,13 @@ test_that("display of successes only is compact", {
8383
test_path("reporters/successes.R")
8484
)
8585
})
86+
87+
# parallel progress reporter ----------------------------------------------
88+
89+
test_that("ParallelProgressReporter fails after max_fail tests", {
90+
withr::local_options(testthat.progress.max_fails = 10)
91+
expect_snapshot_reporter(
92+
ParallelProgressReporter$new(update_interval = 0, min_time = Inf),
93+
test_path(c("reporters/fail-many.R", "reporters/fail.R"))
94+
)
95+
})

0 commit comments

Comments
 (0)