Skip to content

Commit d547cae

Browse files
horenmarhorenmar
committed
Fix bad error reporting for nested exceptions in default configuration
Bad handling of `TestFailureException` when translating unexpected exceptions inside assertion macros led to the unexpected exceptions handling erroring out through throwing the same exception again. This was then backstopped by the machinery for handling uncaught exceptions from assertions, which is normally used by the `CATCH_CONFIG_FAST_COMPILE` machinery, where we assume that it can only be invoked because the assertion macros are not configured to catch assertions. Closes #1292
1 parent fee8162 commit d547cae

21 files changed

+595
-23
lines changed

‎docs/skipping-passing-failing.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,20 @@ TEST_CASE("failing test") {
5252
}
5353
```
5454
55+
Same applies for a `SKIP` nested inside an assertion:
56+
57+
```cpp
58+
static bool do_skip() {
59+
SKIP();
60+
return true;
61+
}
62+
63+
TEST_CASE("Another failing test") {
64+
CHECK(do_skip());
65+
}
66+
```
67+
68+
5569
### Interaction with Sections and Generators
5670

5771
Sections, nested sections as well as specific outputs from [generators](generators.md#top)

‎src/catch2/benchmark/catch_benchmark.hpp‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -86,9 +86,7 @@ namespace Catch {
8686
auto analysis = Detail::analyse(*cfg, samples.data(), samples.data() + samples.size());
8787
BenchmarkStats<> stats{ CATCH_MOVE(info), CATCH_MOVE(analysis.samples), analysis.mean, analysis.standard_deviation, analysis.outliers, analysis.outlier_variance };
8888
getResultCapture().benchmarkEnded(stats);
89-
} CATCH_CATCH_ANON (TestFailureException const&) {
90-
getResultCapture().benchmarkFailed("Benchmark failed due to failed assertion"_sr);
91-
} CATCH_CATCH_ALL{
89+
} CATCH_CATCH_ALL {
9290
getResultCapture().benchmarkFailed(translateActiveException());
9391
// We let the exception go further up so that the
9492
// test case is marked as failed.

‎src/catch2/internal/catch_exception_translator_registry.cpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,10 +59,10 @@ namespace Catch {
5959
// To avoid having to handle TFE explicitly everywhere, we just
6060
// rethrow it so that it goes back up the caller.
6161
catch( TestFailureException& ) {
62-
std::rethrow_exception(std::current_exception());
62+
return "{ nested assertion failed }";
6363
}
6464
catch( TestSkipException& ) {
65-
std::rethrow_exception(std::current_exception());
65+
return "{ nested SKIP() called }";
6666
}
6767
catch( std::exception const& ex ) {
6868
return ex.what();

‎tests/SelfTest/Baselines/automake.sw.approved.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,8 @@ Nor would this
9595
:test-result: PASS Approximate comparisons with mixed numeric types
9696
:test-result: PASS Arbitrary predicate matcher
9797
:test-result: PASS Assertion macros support bit operators and bool conversions
98+
:test-result: XFAIL Assertions can be nested - CHECK
99+
:test-result: XFAIL Assertions can be nested - REQUIRE
98100
:test-result: PASS Assertions then sections
99101
:test-result: PASS Basic use of the Contains range matcher
100102
:test-result: PASS Basic use of the Empty range matcher
@@ -153,6 +155,7 @@ Nor would this
153155
:test-result: PASS Exceptions matchers
154156
:test-result: FAIL Expected exceptions that don't throw or unexpected exceptions fail the test
155157
:test-result: FAIL FAIL aborts the test
158+
:test-result: XFAIL FAIL can be nested in assertion
156159
:test-result: FAIL FAIL does not require an argument
157160
:test-result: FAIL FAIL_CHECK does not abort the test
158161
:test-result: PASS Factorials are computed

‎tests/SelfTest/Baselines/automake.sw.multi.approved.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,8 @@
9393
:test-result: PASS Approximate comparisons with mixed numeric types
9494
:test-result: PASS Arbitrary predicate matcher
9595
:test-result: PASS Assertion macros support bit operators and bool conversions
96+
:test-result: XFAIL Assertions can be nested - CHECK
97+
:test-result: XFAIL Assertions can be nested - REQUIRE
9698
:test-result: PASS Assertions then sections
9799
:test-result: PASS Basic use of the Contains range matcher
98100
:test-result: PASS Basic use of the Empty range matcher
@@ -151,6 +153,7 @@
151153
:test-result: PASS Exceptions matchers
152154
:test-result: FAIL Expected exceptions that don't throw or unexpected exceptions fail the test
153155
:test-result: FAIL FAIL aborts the test
156+
:test-result: XFAIL FAIL can be nested in assertion
154157
:test-result: FAIL FAIL does not require an argument
155158
:test-result: FAIL FAIL_CHECK does not abort the test
156159
:test-result: PASS Factorials are computed

‎tests/SelfTest/Baselines/compact.sw.approved.txt‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -332,6 +332,11 @@ Compilation.tests.cpp:<line number>: passed: !(lhs & rhs) for: !(Val: 1 & Val: 2
332332
Compilation.tests.cpp:<line number>: passed: HasBitOperators{ 1 } & HasBitOperators{ 1 } for: Val: 1 & Val: 1
333333
Compilation.tests.cpp:<line number>: passed: lhs ^ rhs for: Val: 1 ^ Val: 2
334334
Compilation.tests.cpp:<line number>: passed: !(lhs ^ lhs) for: !(Val: 1 ^ Val: 1)
335+
AssertionHandler.tests.cpp:<line number>: failed: i > 10 for: 2 > 10
336+
AssertionHandler.tests.cpp:<line number>: failed: unexpected exception with message: '{ nested assertion failed }'; expression was: foo( 2 ) == 2
337+
AssertionHandler.tests.cpp:<line number>: passed: true
338+
AssertionHandler.tests.cpp:<line number>: failed: i > 10 for: 2 > 10
339+
AssertionHandler.tests.cpp:<line number>: failed: unexpected exception with message: '{ nested assertion failed }'; expression was: foo( 2 ) == 2
335340
Tricky.tests.cpp:<line number>: passed: true
336341
Tricky.tests.cpp:<line number>: passed: true
337342
Tricky.tests.cpp:<line number>: passed: true
@@ -636,6 +641,8 @@ Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'e
636641
Exception.tests.cpp:<line number>: failed: expected exception, got none; expression was: thisDoesntThrow(), std::domain_error
637642
Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'expected exception'; expression was: thisThrows()
638643
Message.tests.cpp:<line number>: failed: explicitly with 1 message: 'This is a failure'
644+
AssertionHandler.tests.cpp:<line number>: failed: explicitly with 1 message: 'Throw a Catch::TestFailureException'
645+
AssertionHandler.tests.cpp:<line number>: failed: unexpected exception with message: '{ nested assertion failed }'; expression was: do_fail()
639646
Message.tests.cpp:<line number>: failed: explicitly
640647
Message.tests.cpp:<line number>: failed: explicitly with 1 message: 'This is a failure'
641648
Message.tests.cpp:<line number>: warning: 'This message appears in the output'
@@ -2862,7 +2869,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
28622869
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
28632870
Misc.tests.cpp:<line number>: passed:
28642871
Misc.tests.cpp:<line number>: passed:
2865-
test cases: 428 | 313 passed | 95 failed | 6 skipped | 14 failed as expected
2866-
assertions: 2281 | 2089 passed | 157 failed | 35 failed as expected
2872+
test cases: 431 | 313 passed | 95 failed | 6 skipped | 17 failed as expected
2873+
assertions: 2288 | 2090 passed | 157 failed | 41 failed as expected
28672874

28682875

‎tests/SelfTest/Baselines/compact.sw.multi.approved.txt‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,11 @@ Compilation.tests.cpp:<line number>: passed: !(lhs & rhs) for: !(Val: 1 & Val: 2
330330
Compilation.tests.cpp:<line number>: passed: HasBitOperators{ 1 } & HasBitOperators{ 1 } for: Val: 1 & Val: 1
331331
Compilation.tests.cpp:<line number>: passed: lhs ^ rhs for: Val: 1 ^ Val: 2
332332
Compilation.tests.cpp:<line number>: passed: !(lhs ^ lhs) for: !(Val: 1 ^ Val: 1)
333+
AssertionHandler.tests.cpp:<line number>: failed: i > 10 for: 2 > 10
334+
AssertionHandler.tests.cpp:<line number>: failed: unexpected exception with message: '{ nested assertion failed }'; expression was: foo( 2 ) == 2
335+
AssertionHandler.tests.cpp:<line number>: passed: true
336+
AssertionHandler.tests.cpp:<line number>: failed: i > 10 for: 2 > 10
337+
AssertionHandler.tests.cpp:<line number>: failed: unexpected exception with message: '{ nested assertion failed }'; expression was: foo( 2 ) == 2
333338
Tricky.tests.cpp:<line number>: passed: true
334339
Tricky.tests.cpp:<line number>: passed: true
335340
Tricky.tests.cpp:<line number>: passed: true
@@ -634,6 +639,8 @@ Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'e
634639
Exception.tests.cpp:<line number>: failed: expected exception, got none; expression was: thisDoesntThrow(), std::domain_error
635640
Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'expected exception'; expression was: thisThrows()
636641
Message.tests.cpp:<line number>: failed: explicitly with 1 message: 'This is a failure'
642+
AssertionHandler.tests.cpp:<line number>: failed: explicitly with 1 message: 'Throw a Catch::TestFailureException'
643+
AssertionHandler.tests.cpp:<line number>: failed: unexpected exception with message: '{ nested assertion failed }'; expression was: do_fail()
637644
Message.tests.cpp:<line number>: failed: explicitly
638645
Message.tests.cpp:<line number>: failed: explicitly with 1 message: 'This is a failure'
639646
Message.tests.cpp:<line number>: warning: 'This message appears in the output'
@@ -2851,7 +2858,7 @@ InternalBenchmark.tests.cpp:<line number>: passed: med == 18. for: 18.0 == 18.0
28512858
InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
28522859
Misc.tests.cpp:<line number>: passed:
28532860
Misc.tests.cpp:<line number>: passed:
2854-
test cases: 428 | 313 passed | 95 failed | 6 skipped | 14 failed as expected
2855-
assertions: 2281 | 2089 passed | 157 failed | 35 failed as expected
2861+
test cases: 431 | 313 passed | 95 failed | 6 skipped | 17 failed as expected
2862+
assertions: 2288 | 2090 passed | 157 failed | 41 failed as expected
28562863

28572864

‎tests/SelfTest/Baselines/console.std.approved.txt‎

Lines changed: 49 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -347,6 +347,38 @@ Exception.tests.cpp:<line number>: FAILED:
347347
due to unexpected exception with message:
348348
unexpected exception
349349

350+
-------------------------------------------------------------------------------
351+
Assertions can be nested - CHECK
352+
-------------------------------------------------------------------------------
353+
AssertionHandler.tests.cpp:<line number>
354+
...............................................................................
355+
356+
AssertionHandler.tests.cpp:<line number>: FAILED:
357+
REQUIRE( i > 10 )
358+
with expansion:
359+
2 > 10
360+
361+
AssertionHandler.tests.cpp:<line number>: FAILED:
362+
CHECK( foo( 2 ) == 2 )
363+
due to unexpected exception with message:
364+
{ nested assertion failed }
365+
366+
-------------------------------------------------------------------------------
367+
Assertions can be nested - REQUIRE
368+
-------------------------------------------------------------------------------
369+
AssertionHandler.tests.cpp:<line number>
370+
...............................................................................
371+
372+
AssertionHandler.tests.cpp:<line number>: FAILED:
373+
REQUIRE( i > 10 )
374+
with expansion:
375+
2 > 10
376+
377+
AssertionHandler.tests.cpp:<line number>: FAILED:
378+
REQUIRE( foo( 2 ) == 2 )
379+
due to unexpected exception with message:
380+
{ nested assertion failed }
381+
350382
-------------------------------------------------------------------------------
351383
Captures do not leave block with an exception
352384
-------------------------------------------------------------------------------
@@ -619,6 +651,21 @@ Message.tests.cpp:<line number>: FAILED:
619651
explicitly with message:
620652
This is a failure
621653

654+
-------------------------------------------------------------------------------
655+
FAIL can be nested in assertion
656+
-------------------------------------------------------------------------------
657+
AssertionHandler.tests.cpp:<line number>
658+
...............................................................................
659+
660+
AssertionHandler.tests.cpp:<line number>: FAILED:
661+
explicitly with message:
662+
Throw a Catch::TestFailureException
663+
664+
AssertionHandler.tests.cpp:<line number>: FAILED:
665+
CHECK_NOTHROW( do_fail() )
666+
due to unexpected exception with message:
667+
{ nested assertion failed }
668+
622669
-------------------------------------------------------------------------------
623670
FAIL does not require an argument
624671
-------------------------------------------------------------------------------
@@ -1672,6 +1719,6 @@ due to unexpected exception with message:
16721719
Why would you throw a std::string?
16731720

16741721
===============================================================================
1675-
test cases: 428 | 331 passed | 76 failed | 7 skipped | 14 failed as expected
1676-
assertions: 2260 | 2089 passed | 136 failed | 35 failed as expected
1722+
test cases: 431 | 331 passed | 76 failed | 7 skipped | 17 failed as expected
1723+
assertions: 2267 | 2090 passed | 136 failed | 41 failed as expected
16771724

‎tests/SelfTest/Baselines/console.sw.approved.txt‎

Lines changed: 52 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2566,6 +2566,41 @@ Compilation.tests.cpp:<line number>: PASSED:
25662566
with expansion:
25672567
!(Val: 1 ^ Val: 1)
25682568

2569+
-------------------------------------------------------------------------------
2570+
Assertions can be nested - CHECK
2571+
-------------------------------------------------------------------------------
2572+
AssertionHandler.tests.cpp:<line number>
2573+
...............................................................................
2574+
2575+
AssertionHandler.tests.cpp:<line number>: FAILED:
2576+
REQUIRE( i > 10 )
2577+
with expansion:
2578+
2 > 10
2579+
2580+
AssertionHandler.tests.cpp:<line number>: FAILED:
2581+
CHECK( foo( 2 ) == 2 )
2582+
due to unexpected exception with message:
2583+
{ nested assertion failed }
2584+
2585+
AssertionHandler.tests.cpp:<line number>: PASSED:
2586+
CHECK( true )
2587+
2588+
-------------------------------------------------------------------------------
2589+
Assertions can be nested - REQUIRE
2590+
-------------------------------------------------------------------------------
2591+
AssertionHandler.tests.cpp:<line number>
2592+
...............................................................................
2593+
2594+
AssertionHandler.tests.cpp:<line number>: FAILED:
2595+
REQUIRE( i > 10 )
2596+
with expansion:
2597+
2 > 10
2598+
2599+
AssertionHandler.tests.cpp:<line number>: FAILED:
2600+
REQUIRE( foo( 2 ) == 2 )
2601+
due to unexpected exception with message:
2602+
{ nested assertion failed }
2603+
25692604
-------------------------------------------------------------------------------
25702605
Assertions then sections
25712606
-------------------------------------------------------------------------------
@@ -4561,6 +4596,21 @@ Message.tests.cpp:<line number>: FAILED:
45614596
explicitly with message:
45624597
This is a failure
45634598

4599+
-------------------------------------------------------------------------------
4600+
FAIL can be nested in assertion
4601+
-------------------------------------------------------------------------------
4602+
AssertionHandler.tests.cpp:<line number>
4603+
...............................................................................
4604+
4605+
AssertionHandler.tests.cpp:<line number>: FAILED:
4606+
explicitly with message:
4607+
Throw a Catch::TestFailureException
4608+
4609+
AssertionHandler.tests.cpp:<line number>: FAILED:
4610+
CHECK_NOTHROW( do_fail() )
4611+
due to unexpected exception with message:
4612+
{ nested assertion failed }
4613+
45644614
-------------------------------------------------------------------------------
45654615
FAIL does not require an argument
45664616
-------------------------------------------------------------------------------
@@ -19134,6 +19184,6 @@ Misc.tests.cpp:<line number>
1913419184
Misc.tests.cpp:<line number>: PASSED:
1913519185

1913619186
===============================================================================
19137-
test cases: 428 | 313 passed | 95 failed | 6 skipped | 14 failed as expected
19138-
assertions: 2281 | 2089 passed | 157 failed | 35 failed as expected
19187+
test cases: 431 | 313 passed | 95 failed | 6 skipped | 17 failed as expected
19188+
assertions: 2288 | 2090 passed | 157 failed | 41 failed as expected
1913919189

‎tests/SelfTest/Baselines/console.sw.multi.approved.txt‎

Lines changed: 52 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2564,6 +2564,41 @@ Compilation.tests.cpp:<line number>: PASSED:
25642564
with expansion:
25652565
!(Val: 1 ^ Val: 1)
25662566

2567+
-------------------------------------------------------------------------------
2568+
Assertions can be nested - CHECK
2569+
-------------------------------------------------------------------------------
2570+
AssertionHandler.tests.cpp:<line number>
2571+
...............................................................................
2572+
2573+
AssertionHandler.tests.cpp:<line number>: FAILED:
2574+
REQUIRE( i > 10 )
2575+
with expansion:
2576+
2 > 10
2577+
2578+
AssertionHandler.tests.cpp:<line number>: FAILED:
2579+
CHECK( foo( 2 ) == 2 )
2580+
due to unexpected exception with message:
2581+
{ nested assertion failed }
2582+
2583+
AssertionHandler.tests.cpp:<line number>: PASSED:
2584+
CHECK( true )
2585+
2586+
-------------------------------------------------------------------------------
2587+
Assertions can be nested - REQUIRE
2588+
-------------------------------------------------------------------------------
2589+
AssertionHandler.tests.cpp:<line number>
2590+
...............................................................................
2591+
2592+
AssertionHandler.tests.cpp:<line number>: FAILED:
2593+
REQUIRE( i > 10 )
2594+
with expansion:
2595+
2 > 10
2596+
2597+
AssertionHandler.tests.cpp:<line number>: FAILED:
2598+
REQUIRE( foo( 2 ) == 2 )
2599+
due to unexpected exception with message:
2600+
{ nested assertion failed }
2601+
25672602
-------------------------------------------------------------------------------
25682603
Assertions then sections
25692604
-------------------------------------------------------------------------------
@@ -4559,6 +4594,21 @@ Message.tests.cpp:<line number>: FAILED:
45594594
explicitly with message:
45604595
This is a failure
45614596

4597+
-------------------------------------------------------------------------------
4598+
FAIL can be nested in assertion
4599+
-------------------------------------------------------------------------------
4600+
AssertionHandler.tests.cpp:<line number>
4601+
...............................................................................
4602+
4603+
AssertionHandler.tests.cpp:<line number>: FAILED:
4604+
explicitly with message:
4605+
Throw a Catch::TestFailureException
4606+
4607+
AssertionHandler.tests.cpp:<line number>: FAILED:
4608+
CHECK_NOTHROW( do_fail() )
4609+
due to unexpected exception with message:
4610+
{ nested assertion failed }
4611+
45624612
-------------------------------------------------------------------------------
45634613
FAIL does not require an argument
45644614
-------------------------------------------------------------------------------
@@ -19123,6 +19173,6 @@ Misc.tests.cpp:<line number>
1912319173
Misc.tests.cpp:<line number>: PASSED:
1912419174

1912519175
===============================================================================
19126-
test cases: 428 | 313 passed | 95 failed | 6 skipped | 14 failed as expected
19127-
assertions: 2281 | 2089 passed | 157 failed | 35 failed as expected
19176+
test cases: 431 | 313 passed | 95 failed | 6 skipped | 17 failed as expected
19177+
assertions: 2288 | 2090 passed | 157 failed | 41 failed as expected
1912819178

0 commit comments

Comments
 (0)