`TestClassOrder` does not play nicely with `@RepeatedTest`

[roded]

Seems like TestClassOrder does not play nicely with @RepeatedTest.

Steps to reproduce

The following results in out of order test execution. I expected all writeTest repetitions to execute before all readTest executions.

Replacing @RepeatedTest with @Test works though.

@TestClassOrder(ClassOrderer.OrderAnnotation.class)
public class RodedIcegenerStressCdsTest {

    @Nested
    @Order(2)
    public class ReadScenarioStressTests {

        // @Test
        @RepeatedTest(10)
        public void readTest() {
            System.out.println("read");
        }
    }

    @Nested
    @Order(1)
    public class WriteScenarioStressTests {

        // @Test
        @RepeatedTest(10)
        public void writeTest() {
            System.out.println("write");
        }
    }
}

Log:

10:04:22.791 INFO  [o.j.j.e.c.EnumConfigurationParameterConverter-ForkJoinPool-1-worker-1] - Using parallel execution mode 'CONCURRENT' set via the 'junit.jupiter.execution.parallel.mode.default' configuration parameter.
read
read
write
write
write
read
read
read
read
read
write
write
write
write
read
write
write
read
write
read

Context

JUnit version: 5.11.3
Running via Gradle on IntelliJ.

    systemProperty("junit.jupiter.execution.parallel.enabled", "true")
    systemProperty("junit.jupiter.execution.parallel.mode.default", "concurrent")
    systemProperty("junit.jupiter.execution.parallel.mode.classes.default", "concurrent")
    systemProperty("junit.jupiter.extensions.autodetection.enabled", "true")
    systemProperty("junit.jupiter.execution.parallel.config.strategy", "fixed")
    systemProperty("junit.jupiter.execution.parallel.config.fixed.parallelism", 10)
    systemProperty("junit.jupiter.execution.parallel.config.fixed.max-pool-size", 10)

[sbrannen]

Hi @roded,

Congratulations on submitting your first issue for JUnit 5! 👍

Replacing @RepeatedTest with @Test works though.

That's actually not true. If you introduce multiple @Test methods in each @Nested test class, you will witness the same behavior that you see with @RepeatedTest.

When you set junit.jupiter.execution.parallel.mode.default = concurrent and junit.jupiter.execution.parallel.mode.classes.default = concurrent, both the test classes and the test methods (or @RepeatedTest invocations) are run currently.

If you want all @Test methods and @RepeatedTest invocations in WriteScenarioStressTests to run first, you should set junit.jupiter.execution.parallel.mode.default = same_thread or simply not configure junit.jupiter.execution.parallel.mode.default (since same_thread is the default).

In light of that, I am closing this issue.

Regards,

Sam

[github-actions]

Please assign a status label to this issue.

[roded]

@sbrannen Thanks for the reply.
I'm trying to be able to keep the high parallelism plus be able to control the order of the tests.
If I understand correctly, same_thread will limit the parallelism considerably.

[sbrannen]

If you run the test classes in parallel, you will still benefit at that level, which can be significant depending on the number of test classes.

If you only want test methods in certain test classes to execute in the same_thread, you can set the default mode to concurrent and selectively annotate individual test classes with @Execution(SAME_THREAD).

Perhaps @marcphilipp can provide additional insight.