Order annotation in parametrized tests not respected with concurrent execution

[PranavMahesh1]

Hi,
I have a class with a couple of parametrized tests (testA, testB). I want to execute the parameters of each test in parallel, but I don't want to execute all the tests at one time. From looking at the documentation, I found we could use @Order annotation. But the order is not being respected - it is concurrently executing both methods instead of doing one method parallel at a time.

MRP:

@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
class ParallelTest {
    @Order(1)
    @ParameterizedTest
    @ValueSource(strings = ["a", "b", "c"])
    @Execution(ExecutionMode.CONCURRENT)
    fun testA(input: String) {
        println("First test with $input")
        Thread.sleep(1000)
    }

    @Order(2)
    @ParameterizedTest
    @ValueSource(strings = ["a", "b", "c"])
    @Execution(ExecutionMode.CONCURRENT)
    fun testB(input: String) {
        println("Second test with $input")
        Thread.sleep(1000)
    }
}

In this example both methods run with input at random order in my IntelliJ output. What I would want is 'First test with input a', b, c, then 'Second test with a', b, c, etc. Also, the order annotation is not working - if I swap the order annotation of both functions it does nothing.

I was able to get what I want somewhat working like this:

@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
class ParallelTest {
    @Nested
    inner class FirstTest {
        @ParameterizedTest
        @ValueSource(strings = ["a", "b", "c"])
        @Execution(ExecutionMode.CONCURRENT)
        fun testA(input: String) {
            println("First test with $input")
            Thread.sleep(1000)
        }
    }

    @Nested
    inner class SecondTest {
        @ParameterizedTest
        @ValueSource(strings = ["a", "b", "c"])
        @Execution(ExecutionMode.CONCURRENT)
        fun testB(input: String) {
            println("Second test with $input")
            Thread.sleep(1000)
        }
    }
}

In this situation, it executes testA in parallel, then moves on to testB. However, the Order annotation doesn't work here either - it seems like it's going off alphabetical order? Even if I add @Order(1) to SecondTest and @Order(2) to FirstTest, it still does FirstTest first. Is the nested solution the only way to get the order I want, and can we not specify the order in this situation? I'm using junit-jupiter-api » 5.9.2.

[marcphilipp]

Using any MethodOrderer with concurrent execution is not supported.

There's this paragraph in the Javadoc:
https://github.com/junit-team/junit5/blob/2fb0e223a7258ee6ed89093ca9095c720d01cb02/junit-jupiter-api/src/main/java/org/junit/jupiter/api/TestMethodOrder.java#L68-L70

It sounds like you'd want something like the target attribute we've recently introduced for @ResourceLock in #3102 so you could write:

@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
class ParallelTest {
    @Order(1)
    @ParameterizedTest
    @ValueSource(strings = ["a", "b", "c"])
    @Execution(value = ExecutionMode.CONCURRENT, target = CHILDREN)
    fun testA(input: String) {
        println("First test with $input")
        Thread.sleep(1000)
    }

    @Order(2)
    @ParameterizedTest
    @ValueSource(strings = ["a", "b", "c"])
    @Execution(value = ExecutionMode.CONCURRENT, target = CHILDREN)
    fun testB(input: String) {
        println("Second test with $input")
        Thread.sleep(1000)
    }
}

That would cause the testA and testB "containers" to be executed sequentially, but their children, i.e. the invocations, concurrently.

[PranavMahesh1]

I see, so it's added to ResourceLock but not Execution at the moment. How about if I don't care about the order, is it still possible to do what I mentioned, besides the nested class approach?

[marcphilipp]

How about if I don't care about the order, is it still possible to do what I mentioned, besides the nested class approach?

No, the nested class approach is currently the only means to accomplish this as far as I'm aware.