When used with JUnit 5, test reports seems incomplete

[ThomasGrt]

👓 What did you see?

When doing sbt test in test report target/test-reports/TEST-bdd.TestValidBDD.xml
I see lots of testcases with same name :

<testcase classname="bdd.TestValidBDD" name="(It is not a test it is a sbt.testing.SuiteSelector)" time="0.298">
<testcase classname="bdd.TestValidBDD" name="(It is not a test it is a sbt.testing.SuiteSelector)" time="0.097">

✅ What did you expect to see?

Previously (JUnit 4) I had detailed names, corresponding to scenario being tested, example :

<testcase classname="bdd.TestValidBDD" name="Authentication.Unauthenticated" time="0.289">
<testcase classname="bdd.TestValidBDD" name="Authentication.Authenticated with an invalid token" time="0.103">

📦 Which tool/library version are you using?

SBT project

sbt plugin :

• addSbtPlugin("com.github.sbt.junit" % "sbt-jupiter-interface" % "0.15.0")

dependencies :

• junit BOM 5.13.4
• "io.cucumber" % "cucumber-junit-platform-engine" % "7.26.0" % Test,
• "com.github.sbt.junit" % "jupiter-interface" % JupiterKeys.jupiterVersion.value % Test,
• "org.junit.jupiter" % "junit-jupiter" % junitBom.key.value % Test,
• "org.junit.platform" % "junit-platform-suite" % junitBom.key.value % Test,

🔬 How could we reproduce it?

Create some scenarios and a Test file like :

import io.cucumber.junit.platform.engine.Constants
import org.junit.platform.suite.api.ConfigurationParameter
import org.junit.platform.suite.api.IncludeEngines
import org.junit.platform.suite.api.SelectPackages
import org.junit.platform.suite.api.Suite

@Suite
@IncludeEngines(Array("cucumber"))
@SelectPackages(Array("features")) // Load all *.feature files in the given package
@ConfigurationParameter(key = Constants.GLUE_PROPERTY_NAME, value = "glue")
@ConfigurationParameter(key = Constants.FILTER_TAGS_PROPERTY_NAME, value = "@valid")
class TestValidBDD {}

launch sbt test

look your test report file in target\test-report

📚 Any additional context?

No response

[mpkorstanje]

@gaeljw while the offending text comes all the way from sbt itself. I think this is another one for sbt-jupiter-interface.

Looking at the code I get the impression that this was created with the assumption that JUnit 5 only consists of JUnit Jupiter and works exactly like JUnit 4. A simple work around like cucumber/cucumber-jvm#3023 will not work here.

@ThomasGrt Cucumber does come with its own JUnit plugin. You can add this to your suite to enable it:

@ConfigurationParameter(
  key = Constants.PLUGIN_PROPERTY_NAME,
  value = "junit:target/cucumber.xml"
)

[gaeljw]

I'm able to reproduce the issue on our own examples project.

Diff JUnit4 vs. 5:

- <testcase classname="cucumber.examples.scalacalculator.RunCukesTest" name="Basic Arithmetic.Adding" time="0.055">
+ <testcase classname="cucumber.examples.scalacalculator.RunCukesTest" name="(It is not a test it is a sbt.testing.SuiteSelector)" time="0.051">

I'll report the issue to sbt-jupiter-interface.

And indeed, using Cucumber's own Junit plugin for reports could be a "workaround".

---

EDIT: reported at sbt/sbt-jupiter-interface#175

[ThomasGrt]

@ThomasGrt Cucumber does come with its own JUnit plugin. You can add this to your suite to enable it:

@ConfigurationParameter(
key = Constants.PLUGIN_PROPERTY_NAME,
value = "junit:target/cucumber.xml"
)

@mpkorstanje
I tested it and it generated a cucumber.xml file but gitlab doesn't seem to take it into account in it's test section.
I think I will wait for the proper fix, no rush on our side.

Anyway Thanks for the help and the quick answer!

[mpkorstanje]

I tested it and it generated a cucumber.xml file but gitlab doesn't seem to take it into account in it's test section.

Ah. Try with "junit:target/test-reports//cucumber.xml" then the files should be in a place where Gitlab can find them.

If that doesn't work, would you mind creating an issue in junit-xml-formatter? The formatter should produce xml files that just work, but the XML schema is only a defacto standard. 😅

[ThomasGrt]

Ah. Try with "junit:target/test-reports//cucumber.xml" then the files should be in a place where Gitlab can find them.

It did the job, thanks for the tip!
I will add this conf until some fix is done in sbt-jupiter-interface (if I understand well)

[mpkorstanje]

Judging by sbt/sbt-jupiter-interface#74 they're very much dependent on pull requests.

[gaeljw]

Yup. I'll give it a try when I have a bit of time.

On the other hand, usage of JUnit in Scala is probably quite low. I may better focus on supporting integration of Cucumber with Scalatest.

Anyways, I don't know much of the internals of any of the two. Gotta learn 😅