Allow local `@BootstrapWith` annotation to override a meta-annotation within the same composed annotation

[snicoll]

Overview

The Javadoc of @BootstrapWith currently states:

This annotation may also be used as a meta-annotation to create custom composed annotations. Note, however, that a locally declared @BootstrapWith annotation (i.e., one that is directly present on the current test class) will override any meta-present declarations of @BootstrapWith.

However, if one tries to declare an annotation at the class level that overrides a declaration that is meta-annotated, this leads to:

Suppressed: java.lang.IllegalStateException: Configuration error: found multiple declarations of @BootstrapWith for test class [com.example.demo.DemoCustomTestBootstrapApplicationTests]: [@org.springframework.test.context.BootstrapWith(value=com.example.demo.CustomTestContextBootstrapper.class), @org.springframework.test.context.BootstrapWith(value=org.springframework.boot.test.context.SpringBootTestContextBootstrapper.class)]
		at org.springframework.test.context.BootstrapUtils.resolveExplicitTestContextBootstrapper(BootstrapUtils.java:193)
		at org.springframework.test.context.BootstrapUtils.resolveTestContextBootstrapper(BootstrapUtils.java:150)
		at org.springframework.test.context.BootstrapUtils.resolveTestContextBootstrapper(BootstrapUtils.java:126)
		at org.springframework.test.context.TestContextManager.<init>(TestContextManager.java:126)
		at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
		at org.springframework.test.context.junit.jupiter.SpringExtension.getTestContextManager(SpringExtension.java:427)
		at org.springframework.test.context.junit.jupiter.SpringExtension.afterAll(SpringExtension.java:181)

With the following declaration:

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@SpringBootTest
@BootstrapWith(CustomTestContextBootstrapper.class)
public @interface CustomTest {
}

I am not sure if the Javadoc is wrong or if the use case above is meant to be supported. Knowing that will help figure out our options for spring-projects/spring-boot#15077.

Related Issues

• Explicit failure if multiple @BootstrapWith annotations are used on the same test [SPR-12602] #17203
• Relax explicit TestContextBootstrapper resolution [SPR-17006] #21544
• Provide module-scoped equivalents to @WebMvcTest and @DataJpaTest moduliths/moduliths#5
• Allow proper replacement of @SpringBootTest and extension of SpringBootTestContextBootstrapper spring-boot#15077

[sbrannen]

Indeed, the Javadoc states that "a locally declared @BootstrapWith annotation (i.e., one that is directly present on the current test class) will override any meta-present declarations of @BootstrapWith."

However, with the example @CustomTest annotation, @BootstrapWith is meta-present on both @CustomTest and @SpringBootTest.

So, in that regard, the Javadoc is correct.

An override is only supported for a @BootstrapWith annotation that is directly present on the actual test class, as can be seen here.

spring-framework/spring-test/src/main/java/org/springframework/test/context/BootstrapUtils.java

Lines 187 to 191 in 68231aa

	// Allow directly-present annotation to override annotations that are meta-present.
	BootstrapWith bootstrapWith = testClass.getDeclaredAnnotation(BootstrapWith.class);
	if (bootstrapWith != null) {
	return bootstrapWith.value();
	}

I am not sure if the Javadoc is wrong or if the use case above is meant to be supported. Knowing that will help figure out our options for spring-projects/spring-boot#15077.

That particular use case with @CustomTest was never intended to be supported.

I suppose it might be possible to support such use cases, but that would require a more involved search algorithm — for example, something that makes use of MergedAnnotation.getDistance() — likely something along the lines of TestPropertySourceUtils.findRepeatableAnnotations().

If that turns out to be a requirement for spring-projects/spring-boot#15077, we can discuss our options further.

[snicoll]

Aha, ok. I think I overlooked the "directly present on the current test class" bit. I rather read directly vs. coming as meta-annotated, sorry.

That particular use case with @CustomTest was never intended to be supported.

OK. Why does it work on the test class then? Looking at the use case that I've referenced, and I am not sure it is valid, I can envision that folks want to override the bootstrapper without having to declare it on each test class. So, from that perspective, I see overriding on the test class more as an escape hatch than anything.

I suppose it might be possible to support such use cases,

Everything is possible, I guess, but I don't want to make things more complicated than they already are. I assume the crux of the problem is that we want to fail hard if there are conflicting options?

If that turns out to be a requirement for spring-projects/spring-boot#15077

It would be interesting to figure that out. Looking at the examples, they'd like to reuse SpringBootTestContextBootstrapper. To be honest, I am not quite sure how that's supposed to work without the test being annotated with @SpringBootTest. The original description starts with:

Since @BootstrapWith annotation can only be present once, any test that want to use the new bootstrap needs to replace , and can not use, @SpringBootTest annotation.

All in all, I think we'd benefit from best practices and guidances as to how custom test setups can be accomplished with the TCF, in particular reusing the bootstrapper implementation.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[sbrannen]

Aha, ok. I think I overlooked the "directly present on the current test class" bit. I rather read directly vs. coming as meta-annotated, sorry.

No worries. The nuances of all of that can be quite confusing.

That particular use case with @CustomTest was never intended to be supported.

OK. Why does it work on the test class then?

That's a bit of a long story, so I'll try to summarize here.

1. I believe we originally simply used the first @BootstrapWith annotation that we found.
2. Then it became apparent that it's problematic if two or more composed annotations declare @BootstrapWith meta-annotations, since we wouldn't really know which one to pick.
3. So, we then added the check to forbid that.
4. Then someone (I believe @odrotbohm in the context of Spring Moduliths) requested that a locally declared @BootstrapWith annotation which is directly present on the test class should override any and all @BootstrapWith meta-annotations.

And that's the status quo.

Looking at the use case that I've referenced, and I am not sure it is valid, I can envision that folks want to override the bootstrapper without having to declare it on each test class.

Yes, I can certainly understand the desire to have it work like that. 👍

So, from that perspective, I see overriding on the test class more as an escape hatch than anything.

Indeed, it's really just an escape hatch.

I suppose it might be possible to support such use cases,

Everything is possible, I guess, but I don't want to make things more complicated than they already are. I assume the crux of the problem is that we want to fail hard if there are conflicting options?

Yes, exactly. The "two conflicting meta-annotations" use case should always be rejected.

But... for the use case you've outlined from the Boot issue, given a @BootstrapWith annotation declared on a custom annotation @X that also happens to be annotated with an annotation @Y that has a @BootstrapWith meta-annotation, I don't think it's unreasonable to expect that the local @BootstrapWith declaration on @X would override the meta-declaration on @Y, especially since the person who wrote the @X annotation is in total control of the annotations they choose to declare on @X.

If that turns out to be a requirement for spring-projects/spring-boot#15077

It would be interesting to figure that out. Looking at the examples, they'd like to reuse SpringBootTestContextBootstrapper. To be honest, I am not quite sure how that's supposed to work without the test being annotated with @SpringBootTest. The original description starts with:

Since @BootstrapWith annotation can only be present once, any test that want to use the new bootstrap needs to replace, and can not use, @SpringBootTest annotation.

My take on that is that they want what you described: local @BootstrapWith overrides a meta-declaration within the same composed annotation.

---

All in all, I think we'd benefit from best practices and guidances as to how custom test setups can be accomplished with the TCF, in particular reusing the bootstrapper implementation.

Along those lines, I just introduced dedicated tests for such scenarios in Framework's test suite. See commit 63723fa.

In that commit, @BootWithBarInsteadOfFoo mimics what those Spring Boot users appear to be attempting.

spring-framework/spring-test/src/test/java/org/springframework/test/context/BootstrapUtilsTests.java

Lines 202 to 214 in 63723fa

	/**
	* This annotation emulates the expectation of certain Spring Boot users who wish
	* to be able to <em>override</em> the {@code @BootstrapWith} meta-annotation
	* declaration on {@code @SpringBootTest} while retaining the other configuration of
	* {@code @SpringBootTest}.
	* <p>In this annotation, {@code @BootWithFoo} plays the role of {@code @SpringBootTest},
	* which the user wishes to override via a local declaration of
	* {@code @BootstrapWith(BarBootstrapper.class)}.
	*/
	@BootWithFoo
	@BootstrapWith(BarBootstrapper.class)
	@Retention(RetentionPolicy.RUNTIME)
	@interface BootWithBarInsteadOfFoo {}

The test for that currently fails, as expected.

However, I realized that users already have a means to achieve that goal via @AliasFor.

spring-framework/spring-test/src/test/java/org/springframework/test/context/BootstrapUtilsTests.java

Lines 216 to 222 in 63723fa

	@BootWithFoo
	@Retention(RetentionPolicy.RUNTIME)
	@interface BootWithConfigurableBootstrapperInsteadOfFoo {
	@AliasFor(annotation = BootstrapWith.class)
	Class<? extends TestContextBootstrapper> value() default BarBootstrapper.class;
	}

The test for that passes.

Of course, by introducing the Class<? extends TestContextBootstrapper> value() default BarBootstrapper.class; attribute in that annotation, users then have the ability to change the bootstrapper within their declaration of @BootWithConfigurableBootstrapperInsteadOfFoo (which I also added a test for), and that may not be desirable.

In any case, @AliasFor can certainly be used as a workaround for the time being. So, I posted that idea on the Boot issue to see if the interested parties find that appealing.