[Tests] Make it possible to run int tests without com.carrotsearch.randomizedtesting.RandomizedRunner.class

[nibix]

Situation

The integration tests which are based on the org.opensearch.test.framework.cluster.LocalCluster currently must use the following test runner and annotations:

@RunWith(com.carrotsearch.randomizedtesting.RandomizedRunner.class)
@ThreadLeakScope(ThreadLeakScope.Scope.NONE)

This is because we have com.carrotsearch.randomizedtesting on the class path; this triggers a special behavior in OpenSearch core:

public final class Randomness {
    private static final Method currentMethod;
    private static final Method getRandomMethod;

    static {
        Method maybeCurrentMethod;
        Method maybeGetRandomMethod;
        try {
            Class<?> clazz = Class.forName("com.carrotsearch.randomizedtesting.RandomizedContext");
            maybeCurrentMethod = clazz.getMethod("current");
            maybeGetRandomMethod = clazz.getMethod("getRandom");
        } catch (Exception e) {
            maybeCurrentMethod = null;
            maybeGetRandomMethod = null;
        }
        currentMethod = maybeCurrentMethod;
        getRandomMethod = maybeGetRandomMethod;
    }

Only the com.carrotsearch.randomizedtesting.RandomizedRunner.class is able to initialize the RandomizedContext. If it is uninitalized, OpenSearch will fail to start.

For core testing, randomized testing might be very useful; for the security int tests, however, it is mostly not used. Still, due to the special core behavior we are forced to use the com.carrotsearch.randomizedtesting.RandomizedRunner.class test runner.

This has the disadvantage that the IDE integration is less than optional; that's especially the case for parameterized tests like the ones introduced in #5632. If we could use the core JUnit test runner, IntelliJ would allow the manual selection of certain test parameter combinations. For the com.carrotsearch.randomizedtesting.RandomizedRunner.class, I can only run the test suite with all of its parameters. For debugging, that's not useful.

Goal

We can fix this with two different approaches:

• Completely remove com.carrotsearch.randomizedtesting from the test classpath. However, we have currently a couple of test implementations which depend on it. These would need to be refactored.
• Create some possibilty to let OpenSearch start if the test runner is not com.carrotsearch.randomizedtesting.RandomizedRunner.class and thus the RandomizedContext is not initialized.

[cwperks]

If we could use the core JUnit test runner, IntelliJ would allow the manual selection of certain test parameter combinations.

I agree that this would be a big benefit. I would like to test parameterized tests have wider adoption across the project as its a simple way to increase coverage.