Add cucumber.junit-platform.discovery.as-root-engine (#3023)

The SBT JUpiter Interface assumes that all tests are classes[1] but
does discovery on the classpath root. As a result, it turns feature
names into class selectors. Which doesn't quite work.

Setting `cucumber.junit-platform.discovery.as-root-engine=false` should
ensure Cucumber only discovers tests when called by another engine (e.g.
the JUnit Platform Suite Engine) and by pass this behaviour.

Author: mpkorstanje

CHANGELOG.md

@@ -10,6 +10,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+### Added
+- [JUnit Platform Engine] Add `cucumber.junit-platform.discovery.as-root-engine` to work around SBT issues ([#3023](https://github.com/cucumber/cucumber-jvm/pull/3023) M.P. Korstanje)
+
 ### Fixed
 - [JUnit Platform Engine] Don't use Java 9+ APIs ([#3025](https://github.com/cucumber/cucumber-jvm/pull/3025) M.P. Korstanje)
 - [JUnit Platform Engine] Implement toString on custom DiscoverySelectors

cucumber-junit-platform-engine/README.md

@@ -37,12 +37,12 @@ erDiagram
 
 In practice, integration is still limited so we discuss the most common workarounds below.
 
-### Maven Surefire and Gradle
+### Maven Surefire, Gradle and SBT
 
 Maven Surefire and Gradle do not yet support discovery of non-class based tests
 (see: [gradle/#4773](https://github.com/gradle/gradle/issues/4773),
-[SUREFIRE-1724](https://issues.apache.org/jira/browse/SUREFIRE-1724)). As a
- workaround, you can either use:
+[maven-surefire/#2065](https://github.com/apache/maven-surefire/issues/2065), [stb-jupiter-interface/#142](https://github.com/sbt/sbt-jupiter-interface/issues/142)).
+As a workaround, you can either use:
  * the [JUnit Platform Suite Engine](https://junit.org/junit5/docs/current/user-guide/#junit-platform-suite-engine);
  * the [JUnit Platform Console Launcher](https://junit.org/junit5/docs/current/user-guide/#running-tests-console-launcher) or;
  * the [Gradle Cucumber-Companion](https://github.com/gradle/cucumber-companion) plugins for Gradle and Maven.
@@ -104,6 +104,19 @@ public class RunCucumberTest {
 }
 ```
 
+##### SBT workarounds
+
+The `sbt-jupiter-interface` assumes that all tests directly under a test engine
+have a class source. This is not the case for Cucumber. By running Cucumber
+indirectly through the JUnit Platform Suite Engine and disabling discovery when
+run directly as a "root engine" this problem is avoided.
+
+Add to `junit-platform.properties`: 
+
+```
+cucumber.junit-platform.discovery.as-root-engine=false
+```
+
 #### Use the JUnit Console Launcher ###
 
 You can integrate the JUnit Platform Console Launcher in your build by using 
@@ -435,9 +448,15 @@ cucumber.filter.tags=                                          # a cucumber tag
 cucumber.glue=                                                 # comma separated package names.
                                                                # example: com.example.glue  
 
+cucumber.junit-platform.discovery.as-root-engine               # true or false
+                                                               # default: true
+                                                               # enable discovery when used as a root engine.
+                                                               # note: Workaround for SBT issues.
+
 cucumber.junit-platform.naming-strategy=                       # long, short or surefire.
                                                                # default: short
-                                                               # include parent descriptor name in test descriptor.
+                                                               # long: include parent descriptor names in test descriptor.
+                                                               # surefire: Workaround to make test names appear nicely with Surefire.
 
 cucumber.junit-platform.naming-strategy.short.example-name=    # number, number-and-pickle-if-parameterized or pickle.
                                                                # default: number-and-pickle-if-parameterized

cucumber-junit-platform-engine/pom.xml

@@ -62,12 +62,12 @@
         </dependency>
         <dependency>
             <groupId>org.junit.jupiter</groupId>
-            <artifactId>junit-jupiter-api</artifactId>
+            <artifactId>junit-jupiter-params</artifactId>
             <scope>test</scope>
         </dependency>
         <dependency>
-            <groupId>org.junit.jupiter</groupId>
-            <artifactId>junit-jupiter-params</artifactId>
+            <groupId>org.junit.platform</groupId>
+            <artifactId>junit-platform-suite</artifactId>
             <scope>test</scope>
         </dependency>
         <dependency>

cucumber-junit-platform-engine/src/main/java/io/cucumber/junit/platform/engine/Constants.java

@@ -207,6 +207,26 @@ public final class Constants {
     @API(status = Status.EXPERIMENTAL, since = "7.16.2")
     public static final String JUNIT_PLATFORM_LONG_NAMING_STRATEGY_EXAMPLE_NAME_PROPERTY_NAME = "cucumber.junit-platform.naming-strategy.long.example-name";
 
+    /**
+     * Property name used to enable discovery as a root engine: {@value}
+     * <p>
+     * Valid values are {@code true}, {@code false}. Default: {@code true}.
+     * <p>
+     * As an engine on the JUnit Platform, Cucumber can participate in discovery
+     * directly as a "root" engine. Or indirectly when used through the JUnit
+     * Platform Suite Engine.
+     * <p>
+     * Some build tools assume that all root engines produce class based tests.
+     * This is not the case for Cucumber. Running Cucumber through the JUnit
+     * Platform Suite Engine. Disabling discovery as a root engine resolves
+     * this.
+     * <p>
+     * Note: If a build tool supports JUnits include/exclude Engine
+     * configuration that option should be preferred over this property.
+     */
+    @API(status = Status.EXPERIMENTAL, since = "7.26.0")
+    public static final String JUNIT_PLATFORM_DISCOVERY_AS_ROOT_ENGINE_PROPERTY_NAME = "cucumber.junit-platform.discovery.as-root-engine";
+
     /**
      * Property name to enable plugins: {@value}
      * <p>
@@ -272,7 +292,7 @@ public final class Constants {
      * <p>
      * Valid values are {@code underscore} or {@code camelcase}.
      * <p>
-     * By defaults are generated using the under score naming convention.
+     * By defaults are generated using the underscore naming convention.
      */
     public static final String SNIPPET_TYPE_PROPERTY_NAME = io.cucumber.core.options.Constants.SNIPPET_TYPE_PROPERTY_NAME;
 

cucumber-junit-platform-engine/src/main/java/io/cucumber/junit/platform/engine/CucumberEngineDescriptor.java

@@ -16,10 +16,6 @@ class CucumberEngineDescriptor extends EngineDescriptor implements Node<Cucumber
     private final CucumberConfiguration configuration;
     private final TestSource source;
 
-    CucumberEngineDescriptor(UniqueId uniqueId, CucumberConfiguration configuration) {
-        this(uniqueId, configuration, null);
-    }
-
     CucumberEngineDescriptor(UniqueId uniqueId, CucumberConfiguration configuration, TestSource source) {
         super(uniqueId, "Cucumber");
         this.configuration = requireNonNull(configuration);

cucumber-junit-platform-engine/src/main/java/io/cucumber/junit/platform/engine/CucumberTestEngine.java

@@ -15,6 +15,7 @@
 import org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutorService;
 
 import static io.cucumber.junit.platform.engine.Constants.FEATURES_PROPERTY_NAME;
+import static io.cucumber.junit.platform.engine.Constants.JUNIT_PLATFORM_DISCOVERY_AS_ROOT_ENGINE_PROPERTY_NAME;
 import static io.cucumber.junit.platform.engine.Constants.PARALLEL_CONFIG_PREFIX;
 import static org.junit.platform.engine.support.discovery.DiscoveryIssueReporter.deduplicating;
 import static org.junit.platform.engine.support.discovery.DiscoveryIssueReporter.forwarding;
@@ -44,28 +45,45 @@ public String getId() {
 
     @Override
     public TestDescriptor discover(EngineDiscoveryRequest discoveryRequest, UniqueId uniqueId) {
-        TestSource testSource = createEngineTestSource(discoveryRequest);
-        CucumberConfiguration configuration = new CucumberConfiguration(discoveryRequest.getConfigurationParameters());
+        ConfigurationParameters configurationParameters = discoveryRequest.getConfigurationParameters();
+        TestSource testSource = createEngineTestSource(configurationParameters);
+        CucumberConfiguration configuration = new CucumberConfiguration(configurationParameters);
         CucumberEngineDescriptor engineDescriptor = new CucumberEngineDescriptor(uniqueId, configuration, testSource);
 
         DiscoveryIssueReporter issueReporter = deduplicating(forwarding( //
             discoveryRequest.getDiscoveryListener(), //
             engineDescriptor.getUniqueId() //
         ));
 
+        // Early out if Cucumber is the root engine and discovery has been
+        // explicitly disabled. Workaround for:
+        // https://github.com/sbt/sbt-jupiter-interface/issues/142
+        if (!supportsDiscoveryAsRootEngine(configurationParameters) && isRootEngine(uniqueId)) {
+            return engineDescriptor;
+        }
+
         FeaturesPropertyResolver resolver = new FeaturesPropertyResolver(new DiscoverySelectorResolver());
         resolver.resolveSelectors(discoveryRequest, engineDescriptor, issueReporter);
         return engineDescriptor;
     }
 
-    private static TestSource createEngineTestSource(EngineDiscoveryRequest discoveryRequest) {
+    private static boolean supportsDiscoveryAsRootEngine(ConfigurationParameters configurationParameters) {
+        return configurationParameters.getBoolean(JUNIT_PLATFORM_DISCOVERY_AS_ROOT_ENGINE_PROPERTY_NAME)
+                .orElse(true);
+    }
+
+    private boolean isRootEngine(UniqueId uniqueId) {
+        UniqueId cucumberRootEngineId = UniqueId.forEngine(getId());
+        return uniqueId.hasPrefix(cucumberRootEngineId);
+    }
+
+    private static TestSource createEngineTestSource(ConfigurationParameters configurationParameters) {
         // Workaround. Test Engines do not normally have test source.
         // Maven does not count tests that do not have a ClassSource somewhere
         // in the test descriptor tree.
         // Gradle will report all tests as coming from an "Unknown Class"
         // See: https://github.com/cucumber/cucumber-jvm/pull/2498
-        ConfigurationParameters configuration = discoveryRequest.getConfigurationParameters();
-        if (configuration.get(FEATURES_PROPERTY_NAME).isPresent()) {
+        if (configurationParameters.get(FEATURES_PROPERTY_NAME).isPresent()) {
             return ClassSource.from(CucumberTestEngine.class);
         }
         return null;

cucumber-junit-platform-engine/src/test/java/io/cucumber/junit/platform/engine/CucumberTestEngineTest.java

@@ -20,6 +20,9 @@
 import org.junit.platform.engine.support.descriptor.FileSource;
 import org.junit.platform.engine.support.hierarchical.ExclusiveResource;
 import org.junit.platform.engine.support.hierarchical.Node;
+import org.junit.platform.suite.api.IncludeEngines;
+import org.junit.platform.suite.api.SelectClasspathResource;
+import org.junit.platform.suite.api.Suite;
 import org.junit.platform.testkit.engine.EngineDiscoveryResults;
 import org.junit.platform.testkit.engine.EngineTestKit;
 import org.junit.platform.testkit.engine.Event;
@@ -44,6 +47,7 @@
 import static io.cucumber.junit.platform.engine.Constants.FEATURES_PROPERTY_NAME;
 import static io.cucumber.junit.platform.engine.Constants.FILTER_NAME_PROPERTY_NAME;
 import static io.cucumber.junit.platform.engine.Constants.FILTER_TAGS_PROPERTY_NAME;
+import static io.cucumber.junit.platform.engine.Constants.JUNIT_PLATFORM_DISCOVERY_AS_ROOT_ENGINE_PROPERTY_NAME;
 import static io.cucumber.junit.platform.engine.Constants.JUNIT_PLATFORM_LONG_NAMING_STRATEGY_EXAMPLE_NAME_PROPERTY_NAME;
 import static io.cucumber.junit.platform.engine.Constants.JUNIT_PLATFORM_NAMING_STRATEGY_PROPERTY_NAME;
 import static io.cucumber.junit.platform.engine.Constants.JUNIT_PLATFORM_SHORT_NAMING_STRATEGY_EXAMPLE_NAME_PROPERTY_NAME;
@@ -93,6 +97,17 @@ class CucumberTestEngineTest {
 
     private final CucumberTestEngine engine = new CucumberTestEngine();
 
+    private static Set<UniqueId> discoverUniqueIds(DiscoverySelector discoverySelector) {
+        return EngineTestKit.engine(ENGINE_ID)
+                .selectors(discoverySelector)
+                .execute()
+                .allEvents()
+                .map(Event::getTestDescriptor)
+                .filter(Predicate.not(TestDescriptor::isRoot))
+                .map(TestDescriptor::getUniqueId)
+                .collect(toSet());
+    }
+
     @Test
     void id() {
         assertEquals(ENGINE_ID, engine.getId());
@@ -460,17 +475,6 @@ void supportsUniqueIdSelectorCachesParsedFeaturesAndPickles() {
         assertEquals(pickleIdsFromFeature, pickleIdsFromPickles);
     }
 
-    private static Set<UniqueId> discoverUniqueIds(DiscoverySelector discoverySelector) {
-        return EngineTestKit.engine(ENGINE_ID)
-                .selectors(discoverySelector)
-                .execute()
-                .allEvents()
-                .map(Event::getTestDescriptor)
-                .filter(Predicate.not(TestDescriptor::isRoot))
-                .map(TestDescriptor::getUniqueId)
-                .collect(toSet());
-    }
-
     @Test
     void supportsFilePositionFeature() {
         EngineTestKit.engine(ENGINE_ID)
@@ -604,6 +608,42 @@ void onlySetsEngineSourceWhenFeaturesPropertyIsUsed() {
                 .haveExactly(1, event(test(finishedSuccessfully())));
     }
 
+    @Suite
+    @IncludeEngines("cucumber")
+    @SelectClasspathResource("io/cucumber/junit/platform/engine/single.feature")
+    static class SuiteTestCase {
+
+    }
+
+    @Test
+    void supportsDisablingDiscoveryAsRootEngine() {
+        DiscoverySelector selector = selectClasspathResource("io/cucumber/junit/platform/engine/single.feature");
+
+        // Ensure classpath resource exists.
+        assertThat(EngineTestKit.engine(ENGINE_ID)
+                .selectors(selector)
+                .discover()
+                .getEngineDescriptor()
+                .getChildren())
+                .isNotEmpty();
+
+        assertThat(EngineTestKit.engine(ENGINE_ID)
+                .configurationParameter(JUNIT_PLATFORM_DISCOVERY_AS_ROOT_ENGINE_PROPERTY_NAME, "false")
+                .selectors(selector)
+                .discover()
+                .getEngineDescriptor()
+                .getChildren())
+                .isEmpty();
+
+        assertThat(EngineTestKit.engine("junit-platform-suite")
+                .configurationParameter(JUNIT_PLATFORM_DISCOVERY_AS_ROOT_ENGINE_PROPERTY_NAME, "false")
+                .selectors(selectClass(SuiteTestCase.class))
+                .discover()
+                .getEngineDescriptor()
+                .getChildren())
+                .isNotEmpty();
+    }
+
     @Test
     void selectAndSkipDisabledScenarioByTags() {
         EngineTestKit.engine(ENGINE_ID)