Allow a global option to supress messages when using testthat

[venpopov]

I'd like to reopen #1746. I understand that in testthat 3e it is deliberate that all messages are printed when testthat runs, and that I can handle individual messages with "supressMessages" around the function I am testing.

I find this inconvinient, especially if you suddenly add a message that affects a lot of existing tests, and then you manually have to go and put supressMessages around each individual test. This also makes the testthat code more difficult to read.

[hydrox24]

I think there's a strong case to make suppressing messages the default, and if not by default, then with an option. In particular, messages should be captured and shown on failure, or discarded on success. Hadley argues in the release notes for the third edition that:

For reasons that I can no longer remember, testthat silently ignores all messages. This is inconsistent with other types of output, so as of the 3rd edition, they now bubble up to your test results. (source)

I'm not convinced that messages should be handled similarly to other output, since they are the interface by which testthat communicates results clearly to the user. By way of example, I think good unit testing libraries provide a feature to suppress printing or do it by default. For instance (from the python world):

• Nose will capture (suppress) printed messages and only show them on failure. (source)
• Unittest provides the -b flag which will capture output (stdout and stderr) and only display it on error.

I'll note here that this issue also re-raises #189 too.

[venpopov]

And sometimes messages come from other packages functions called from our functions that I have no interest in seeing in my testing output.

@hydrox24 suggestion to have all output captured and displayed only if tests fail is a great option.

What is the reasoning that messages should be handled on a case by case basis? They are information for users, not relevant for testing unless we are specifically testing if a message appears.

[hadley]

One reason to not suppress messages is that you need some way to get output out of a test and directly to the user's eyes for when you're debugging a test that works interactively.

I don't generally find suppressing messages to be very arduous, so I find it hard to justify making such a big change to the behaviour of testthat.

[GildasLepennetier]

Most of the functions in testthat are already silent when there is no error.

expect_equal(0,0)

expect_equal(0,1)
Error: 0 not equal to 1.
1/1 mismatches
[1] 0 - 1 == -1

The main reason I am using test_that() is to annotate why I am doing this test. It is basically annotation that help A LOT on the long run. I think we agree that there is a big difference between:

• case 1 - without test_that()

x=1;y=0
expect_equal(x,y)
Error: x not equal to y.
1/1 mismatches
[1] 1 - 0 == 1

• case 2 - with test_that() = more context and annotation of the code

x=1;y=0
test_that("The value provided by the client and stored in x is not valid", expect_equal(x,y))

── Failure: The value provided by the client and stored in x is not valid ────────
x not equal to y.
1/1 mismatches
<[1] 1 - 0 == 1

BUT, when there is no error, the "success" message for the test is just spam for some people that uses a lot of tests.

I believe the success output should be optional, and a verbose_success = T / F should be available.

[hadley]

The principle that testthat follows today is consistent — all conditions (messages, warnings, and errors) bubble up, unless otherwise caught. Similarly, all output, whether to stdout() or stderr(), is also pass through to the user (even though it might make the reporter look weird).

So I'm not at all convinced we should make messages silent; it would make their handling inconsistent with all other outputs and I believe would make testthat overall more confusing.

(And if messages are a pain point for your tests, there are already ways to handle them, such as only emitting when !is_testing())