Stop referring to outdated milestone versions of Maven Surefire/Failsafe in the User Guide

[jean-andre-gauthier]

Hi team, I noticed that tests were not executed anymore in older surefire versions (i.e. 2.22.2-3.0.0-M3), when upgrading to 5.12.0-M1.

For surefire 2.22.2 and 3.0.0-M3 we get this warning message, but in 3.0.0-M1 and 3.0.0-M2 the tests are silently ignored:

WARNING: TestEngine with ID 'junit-jupiter' failed to discover tests
org.junit.platform.commons.JUnitException: OutputDirectoryProvider not available; probably due to unaligned versions of the junit-platform-engine and junit-platform-launcher jars on the classpath/module path.

Was this an intentional change / does JUnit 5.12.0 drop support for those surefire versions?

Steps to reproduce

Run ./mvnw clean test in the reproducer. For 5.11.4 the build fails, but for 5.12.0-M1 the build succeeds, because no tests are executed:

reproducer.zip

[sbrannen]

Was this an intentional change / does JUnit 5.12.0 drop support for those surefire versions?

The JUnit team does not provide support for Maven Surefire, since that is an independent, third-party project.

To make your build work, you need to register the correct junit-platform-launcher dependency or use Maven Surefire/Failsafe 3.0.0-M4.

This is documented in the JUnit User Guide.

In light of that, I am closing this issue.

[marcphilipp]

@jean-andre-gauthier Thank you for testing the milestone and providing a reproducer!

[khmarbaise]

Is there a good reason to use old surefire/failsafe plugin versions instead of the most recent versions https://maven.apache.org/plugins/ At the moment surefire/failsafe version is 3.5.2 ? The version 2.22.2 is from 2019 while 3.0.0-M3 is from 2018...
In the earlier versions the automatic identifcation to use a particluar engine... the JUnit-Jupiter-Engine has not been correct. Using the most recent version 3.5.2 will execute the tests correctly and fail as intended...

[INFO] --- compiler:3.13.0:testCompile (default-testCompile) @ maven-build-scan-quickstart ---
[INFO] Recompiling the module because of changed source code.
[INFO] Compiling 2 source files with javac [debug target 1.8] to target/test-classes
[WARNING] bootstrap class path not set in conjunction with -source 8
[INFO] 
[INFO] --- surefire:3.5.2:test (default-test) @ maven-build-scan-quickstart ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running example.TestB
[INFO] [stdout] testB1
[INFO] [stdout] testB2
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.027 s <<< FAILURE! -- in example.TestB
[ERROR] example.TestB.testB1 -- Time elapsed: 0.014 s <<< FAILURE!
org.opentest4j.AssertionFailedError: testB1 fail
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
	at example.TestB.testB1(TestB.java:8)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[INFO] Running example.TestA
[INFO] [stdout] testA1
[INFO] [stdout] testA2
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.003 s -- in example.TestA
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestB.testB1:8 testB1 fail
[INFO] 
[ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 1
[INFO] 
[INFO] --------------------------------------------------------------------------------------------------------------------------
[INFO] BUILD FAILURE

[jean-andre-gauthier]

The failure got picked up in a couple of our automated tests that were testing older surefire versions when 5.12.0-M1 got released, so I was unsure if this was a regression. There's no specific reason for using those older surefire versions beyond that, and adding the junit-platform-launcher dependency solved the issue. Thanks all again for looking into this

[sbrannen]

Hi @khmarbaise,

Is there a good reason to use old surefire/failsafe plugin versions instead of the most recent versions maven.apache.org/plugins

No, definitely not.

And I apologize: I just realized I didn't type "or later" in my comment -- as in, "use Maven Surefire/Failsafe 3.0.0-M4 or later."

At the moment surefire/failsafe version is 3.5.2 ? The version 2.22.2 is from 2019 while 3.0.0-M3 is from 2018...

Good point. 👍

We should actually proactively mention in the User Guide that Surefire and Failsafe have moved far beyond the 3.0 milestone phase.

I am therefore reopening this issue and repurposing it to improve the documentation.