Update our tests to use JUnit 5

[gaeljw]

🤔 What's the problem you've observed?

As Cucumber 7.24.0 deprecated cucumber-junit (JUnit 4), we should update our examples and integration tests to rely on JUnit 5.

Documentation should reflect this as well.

This will allow to merge #398 (even though we could probably upgrade by adding some ignore flags).

✨ Do you have a proposal for making it better?

No response

📚 Any additional context?

No response

[gaeljw]

When trying to switch the examples project to use JUnit 5 with cucumber-junit-platform-engine, I get a weird exception like if JUnit is trying to load the feature file as a class:

org.junit.platform.commons.JUnitException: TestEngine with ID 'cucumber' failed to discover tests
  | => eat org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:208)
        at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverSafely(EngineDiscoveryOrchestrator.java:174)
        at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:119)
        at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:84)
        at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:104)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:91)
        at org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47)
        at org.junit.platform.launcher.core.InterceptingLauncher.lambda$execute$1(InterceptingLauncher.java:39)
        at org.junit.platform.launcher.core.ClasspathAlignmentCheckingLauncherInterceptor.intercept(ClasspathAlignmentCheckingLauncherInterceptor.java:25)
        at org.junit.platform.launcher.core.InterceptingLauncher.execute(InterceptingLauncher.java:38)
        at org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47)
        at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:66)
        at com.github.sbt.junit.jupiter.internal.JupiterRunner$JupiterTaskExecutor.execute(JupiterRunner.java:184)
        at com.github.sbt.junit.jupiter.internal.JupiterRunner$WithCustomProperties.execute(JupiterRunner.java:236)
        at com.github.sbt.junit.jupiter.internal.JupiterRunner$JupiterTask.execute(JupiterRunner.java:123)
        at sbt.TestRunner.runTest$1(TestFramework.scala:153)
        at sbt.TestRunner.run(TestFramework.scala:168)
        at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.$anonfun$apply$1(TestFramework.scala:336)
        at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:296)
        at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
        at sbt.TestFramework$$anon$3$$anonfun$$lessinit$greater$1.apply(TestFramework.scala:336)
        at sbt.TestFunction.apply(TestFramework.scala:348)
        at sbt.Tests$.$anonfun$toTask$1(Tests.scala:436)
        at sbt.std.Transform$$anon$3.$anonfun$apply$2(Transform.scala:47)
        at sbt.std.Transform$$anon$4.work(Transform.scala:69)
        at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
        at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
        at sbt.Execute.work(Execute.scala:292)
        at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
        at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.junit.platform.commons.PreconditionViolationException: Could not load class with name: Basic Arithmetic
        at org.junit.platform.engine.discovery.ClassSelector.lambda$getJavaClass$0(ClassSelector.java:98)
        at org.junit.platform.commons.function.Try$Failure.getOrThrow(Try.java:335)
        at org.junit.platform.engine.discovery.ClassSelector.getJavaClass(ClassSelector.java:97)
        at io.cucumber.junit.platform.engine.FeatureResolver.resolveClass(FeatureResolver.java:180)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at io.cucumber.junit.platform.engine.DiscoverySelectorResolver.resolve(DiscoverySelectorResolver.java:82)
        at io.cucumber.junit.platform.engine.DiscoverySelectorResolver.resolveSelectors(DiscoverySelectorResolver.java:53)
        at io.cucumber.junit.platform.engine.CucumberTestEngine.discover(CucumberTestEngine.java:47)
        at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:195)
        ... 36 more
Caused by: java.lang.ClassNotFoundException: Basic Arithmetic
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
        at sbt.internal.ManagedClassLoader.findClass(ManagedClassLoader.java:103)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:534)
        at java.base/java.lang.Class.forName(Class.java:513)
        at org.junit.platform.commons.util.ReflectionUtils.lambda$tryToLoadClass$10(ReflectionUtils.java:886)
        at org.junit.platform.commons.function.Try.lambda$call$0(Try.java:57)
        at org.junit.platform.commons.function.Try.of(Try.java:93)
        at org.junit.platform.commons.function.Try.call(Try.java:57)
        at org.junit.platform.commons.util.ReflectionUtils.tryToLoadClass(ReflectionUtils.java:873)
        at org.junit.platform.commons.util.ReflectionUtils.tryToLoadClass(ReflectionUtils.java:810)
        at org.junit.platform.commons.support.ReflectionSupport.tryToLoadClass(ReflectionSupport.java:95)
        at org.junit.platform.engine.discovery.ClassSelector.getJavaClass(ClassSelector.java:95)
        ... 42 more
[info] Test run started (JUnit Platform Suite)
[info] Test classpath:cucumber/examples/scalacalculator/basic_arithmetic.feature 4 started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 0.117s
[error] Error: Total 2, Failed 0, Errors 1, Passed 1
[error] Error during tests:
[error]         Basic Arithmetic
[error] (Test / test) sbt.TestsFailedException: Tests unsuccessful

I tried a couple of variations of the following setup but nothing worked so far:

@Suite
@IncludeEngines(Array("cucumber"))
@SelectClasspathResource("cucumber/examples/scalacalculator")
@SelectPackages(Array("cucumber.examples.scalacalculator"))
@ConfigurationParameter(
  key = Constants.GLUE_PROPERTY_NAME,
  value = "cucumber.examples.scalacalculator"
)
//@ConfigurationParameter(
//  key = Constants.FEATURES_PROPERTY_NAME,
//  value = "examples/src/test/resources/cucumber/examples/scalacalculator"
//)
class RunCukesTest

The weird thing is that it says 1 tests passed and 1 errored wheras there is only 1 test. Like there's 2 mechanisms trying to run the tests?

[gaeljw]

Poking @mpkorstanje if you've got any idea where to look at as I think you worked on the JUnit stuff in cucumber-jvm. Don't spend much time on this, it's only if it reminds you anything you've seeen previously 🤷

[mpkorstanje]

The weird thing is that it says 1 tests passed and 1 errored wheras there is only 1 test. Like there's 2 mechanisms trying to run the tests?

That seems extremely likely. Do you have a branch for me to look at?

[mpkorstanje]

If anything you shouldn't use both @SelectPackages(Array("cucumber.examples.scalacalculator")) and @SelectClasspathResource("cucumber/examples/scalacalculator"). The former is for all features in a package, the latter for individual feature files.

[mpkorstanje]

And it looks like a class selector was created for the class Basic Arithmetic. Which is a string that occurs nowhere outside of one feature file. Which is weird because selectors are provided to Junit and then Cucumber by the SBT (presumably).

And tracing that we end up here:

https://github.com/sbt/sbt-jupiter-interface/blob/67f99dbad1a8c687f5ad76bb2d1482f1dac04099/src/library/src/main/java/com/github/sbt/junit/jupiter/internal/JupiterRunner.java#L157

So presumably you've created an SBT suite with the name Basic Arithmetic, and that should probably be RunCukesTest.

[mpkorstanje]

Ah!

https://github.com/sbt/sbt-jupiter-interface/tree/67f99dbad1a8c687f5ad76bb2d1482f1dac04099?tab=readme-ov-file#jupiter-interface

sbt-jupiter-interface: sbt plugin. The sbt plugin makes the runtime library available to the sbt build by adding it to sbt.Keys.testFrameworks. It also overwrites sbt.Keys.detectTests with a custom task that uses JUnits discovery mechanism to collect available tests. This step is necessary since sbt is currently not capable of detecting package private test classes.

Looks like the SBT plugin also does test discovery, but then mucks it up.

So that is probably what they're doing wrong here:

https://github.com/sbt/sbt-jupiter-interface/blob/67f99dbad1a8c687f5ad76bb2d1482f1dac04099/src/library/src/main/java/com/github/sbt/junit/jupiter/api/JupiterTestCollector.java#L281

[mpkorstanje]

And yes! sbt/sbt-jupiter-interface#142

[gaeljw]

Wow, that was fast, thanks a lot @mpkorstanje ! I will follow up on the SBT side of things then :)

[mpkorstanje]

If that doesn't work out, could you try this solution cucumber/cucumber-jvm#3023?

[gaeljw]

Sure, I'll try this tomorrow :)