Javadoc of `@BootstrapWith` implies that multiple instances on the same test class are possible

[snicoll]

The Javadoc of BoostrapWith currently states:

This annotation may also be used as a meta-annotation to create custom composed annotations. Note, however, that a locally declared @BootstrapWith annotation (i.e., one that is directly present on the current test class) will override any meta-present declarations of @BootstrapWith.

However, if one tries to declare an annotation at class level that override a declaration that is meta-annotated, this leads to:

Suppressed: java.lang.IllegalStateException: Configuration error: found multiple declarations of @BootstrapWith for test class [com.example.demo.DemoCustomTestBootstrapApplicationTests]: [@org.springframework.test.context.BootstrapWith(value=com.example.demo.CustomTestContextBootstrapper.class), @org.springframework.test.context.BootstrapWith(value=org.springframework.boot.test.context.SpringBootTestContextBootstrapper.class)]
		at org.springframework.test.context.BootstrapUtils.resolveExplicitTestContextBootstrapper(BootstrapUtils.java:193)
		at org.springframework.test.context.BootstrapUtils.resolveTestContextBootstrapper(BootstrapUtils.java:150)
		at org.springframework.test.context.BootstrapUtils.resolveTestContextBootstrapper(BootstrapUtils.java:126)
		at org.springframework.test.context.TestContextManager.<init>(TestContextManager.java:126)
		at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
		at org.springframework.test.context.junit.jupiter.SpringExtension.getTestContextManager(SpringExtension.java:427)
		at org.springframework.test.context.junit.jupiter.SpringExtension.afterAll(SpringExtension.java:181)

With the following declaration:

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@SpringBootTest
@BootstrapWith(CustomTestContextBootstrapper.class)
public @interface CustomTest {
}

I am not sure if the Javadoc is wrong or if the use case above is meant to be supported. Knowing that will help figure out our options for spring-projects/spring-boot#15077.

[sbrannen]

Indeed, the Javadoc states that "a locally declared @BootstrapWith annotation (i.e., one that is directly present on the current test class) will override any meta-present declarations of @BootstrapWith."

However, with the example @CustomTest annotation, @BootstrapWith is meta-present on both @CustomTest and @SpringBootTest.

So, in that regard, the Javadoc is correct.

An override is only supported for a @BootstrapWith annotation that is directly present on the actual test class, as can be seen here.

spring-framework/spring-test/src/main/java/org/springframework/test/context/BootstrapUtils.java

Lines 187 to 191 in 68231aa

	// Allow directly-present annotation to override annotations that are meta-present.
	BootstrapWith bootstrapWith = testClass.getDeclaredAnnotation(BootstrapWith.class);
	if (bootstrapWith != null) {
	return bootstrapWith.value();
	}

I am not sure if the Javadoc is wrong or if the use case above is meant to be supported. Knowing that will help figure out our options for spring-projects/spring-boot#15077.

That particular use case with @CustomTest was never intended to be supported.

I suppose it might be possible to support such use cases, but that would require a more involved search algorithm — for example, something that makes use of MergedAnnotation.getDistance() — likely something along the lines of TestPropertySourceUtils.findRepeatableAnnotations().

If that turns out to be a requirement for spring-projects/spring-boot#15077, we can discuss our options further.

[snicoll]

Aha, ok. I think I overlooked the "directly present on the current test class" bit. I rather read directly vs. coming as meta-annotated, sorry.

That particular use case with @customTest was never intended to be supported.

OK. Why does it work on the test class then? Looking at the use case that I've referenced, and I am not sure it is valid, I can envision that folks want to override the bootstrapper without having to declare it on each test class. So, from that perspective, I see overriding on the test class more as an escape hatch than anything.

I suppose it might be possible to support such use cases,

Everything is possible, I guess, but I don't want to make things more complicated than they already are. I assume the crux of the problem is that we want to fail hard if there are conflicting options?

If that turns out to be a requirement for spring-projects/spring-boot#15077

It would be interesting to figure that out. Looking at the examples, they'd like to reuse SpringBootTestContextBootstrapper. To be honest, I am not quite sure how that's supposed to work without the test being annotated with @SpringBootTest. The original description starts with:

Since @BootstrapWith annotation can only be present once, any test that want to use the new bootstrap needs to replace , and can not use, @SpringBootTest annotation.

All in all, I think we'd benefit from best practices and guidances as to how custom test setups can be accomplished with the TCF, in particular reusing the bootstrapper implementation.