[Junit5 extension] Run test multiple times for the same flag with different values

[aepfli]

Sometimes, you want to ensure that your code behaves the same way for all code paths, even if different feature flags are involved.

Junit5 has parameterized tests to run multiple tests based on a source. This approach is not feasible for us, as it only generates an array of possible argument sets.

We need to find a way (maybe with our annotation) to create such tests. The cartesian test extension of JUnit Pioneer can be an inspiration for this https://github.com/junit-pioneer/junit-pioneer/blob/main/src/main/java/org/junitpioneer/jupiter/cartesian/CartesianTest.java

[aepfli]

theory crafting:

@FlagTest( name = "<flagName>", values = {"true", "false"}) 

[toddbaert]

@UtkarshSharma2612 you might like this.

[UtkarshSharma2612]

Sure let me check this

[UtkarshSharma2612]

Hey @aepfli I understood the requirement a bit, I wanted to know do we want to change the existing testcases or write new testcases for all the providers with this custom annotation.

[aepfli]

It is about adding another Test Annotation, so people who want to write unit tests to test all kinds of variations with one feature flag can use it. maybe some people want to verify, that the code works the same in both cases and is not effected.

eg. currently people would need to write two tests for a boolean to check if the code behaves as expected:

@Test
@Flag(name = "BOOLEAN_FLAG", value = "true")
void trueCase() {
    // your test code
}

@Test
@Flag(name = "BOOLEAN_FLAG", value = "false")
void falseCase() {
    // your test code
}

instead of this they code do

@FlagTest(name = "BOOLEAN_FLAG", values = { "true", "false" })
void bothCases() {
    // this block will run twice, once for 'true' and once for 'false'
}

does this make somewhat sense? - this is only about testing openFeature functionality with junit

[UtkarshSharma2612]

Okay so we want to create a annotation or something that takes two parameters name (string) and values (List) and runs test cases for all the values. Correct me if I am wrong.

[aepfli]

if you do need more guidance or pointers let me know, some people like the challenge of figuring things out on their own, some want more pointers, and i don't want to spoil the fun for you, if you like to dig on your own ;)

[aepfli]

#1091 seems to have set the basics - we could think about a flagsource for the parameterized tests ;)

[liran2000]

hi @aepfli ,

First of all, I found this extension very useful, and already adopted it to our CI using Flag annotation.

Recently, I also found the need to run test multiple times for the same flag with different values, as described in this issue, expecting something like:

@flagtest( name = "flagName", values = {"true", "false"})

I tried to understand whether and how is this supported now via #1093 , and cannot figure it out, also didn't found such example at the readme.

Can you help clarifying it ?

[aepfli]

It just provides a basic setup for parameterized tests. I still think implementation work is needed, but I have not had the time to look into this.

The added functionality for the template is the basis. but we still need to define an annotation, parse the content, and change the execution. But i think there is still more investigation needed for this scenario. So far i am sadly occupied with other topic, and do not have the time to investigate further.

[sebastian-zahrhuber]

Hi @liran2000 ,
#1093 is different than this issue and does not solve the issue with feature flag values beeing parameters for parameterized tests.
#1093 just implements support for the general use of parameterized tests, but not with feature flag values being the parameters.