Events do not get observed, in multiple scenario tests

[Lausi95]

I have a fairly simple repository with a number of application module tests.
These tests trigger a an event and expect another event to appear. When I run each test in isolation, they work fine. But when I run the whole test suite at once, some, seemingly random, scenarios won't catch the published events.
In the logs, you can see that they are published:
The Initial event got triggered,
then the MisterxErrorEvent is triggered. Therefore it should be seen by the scenario. But as you can see, it threw an exception because it seemingly did not receive the exception.

This is the test in question:

The test names were named to test because i wanted to find out what might be the problem, but i could not find anything on my side that could be off.

Cherry on top: When I deactivate some tests, OTHER tests fail.

The repository is here:
https://github.com/Lausi95/misterx

[raoulvdberge]

I have the exact same issue:

• Publishing event A triggers event B
• The assertion that event A triggers event B works in isolation, but not with other tests
• The "other tests" that break the assertion don't make any sense. For example, the assertion breaks after more than 3 other tests run along.

I've been trying to make a minimal reproduction case, but I can't seem to reproduce it in a test project.

Since my minimal reproduction setup is not working, I'll try to find the cause of this bug with debugging within Spring.

[raoulvdberge]

OK, after some debugging the broken state:

I put a conditional breakpoint in multicastEvent breaking on event B (the event that cannot be verified).

You can see DefaultPublishedEvents listener 18808.
Then, later, back in the test where I assert, notice that the DefaultPublishedEvents is different here (listener 23703):

Now, I put the test suite in a state where the tests will pass again (by reducing the amount of tests from 5 to 4):

The DefaultPublishedEvents instance is different yet again, but this time the tests pass.

At first I thought the different instance is the cause of this bug, but doesn't seem to be the case.

[raoulvdberge]

OK. Another comparison, looking at the PublishedEventsParameterResolver thread delegate. The tests are now in a passing state:

Getting DefaultPublishedEvents that corresponds to the thread with breakpoint watch ((PublishedEventsParameterResolver.ThreadBoundApplicationListenerAdapter) ((ImmutableCollections.ListN) listeners).get(4)).delegate.get() = instance 22695

Then in the verification, instance 22695

Now back to the failing state (increased the amount of tests):

Instance 22800

Verification, instance 22823, this one differs!

In the succeeding case, the instance of the DefaultPublishedEvents thread local is the same as the Scenario one.

In the failing case, the instance of the DefaultPublishedEvents thread local is different from the Scenario one.

[raoulvdberge]

Looking at the failing case, the event never seems to reach the DefaultPublishedEvents because there isn't a listener registered for the thread (looking in ThreadBoundApplicationListenerAdapter)

Breakpoint condition event instanceof PayloadApplicationEvent<?> p && p.payload instanceof AssignmentRemovedEvent && listener == null

In the succeeding case, the breakpoint is never hit.

In the succeeding case, the task thread (the one where @ApplicationModuleListener runs in) inherits the ApplicationListener<ApplicationEvent>, but in the failing case, the task thread didn't inherit anything.

Unsure to me why that is, and why it happens when playing with the amount of tests being ran. The spring web threads and task threads should inherit from the test's thread and InheritableThreadLocal should always be able to resolve a listener, but in some cases (where there are a lot of tests) it doesn't seem to be able to resolve.

Overall, I'm not sure how robust the ThreadBoundApplicationListenerAdapter is. It seems to be unreliable, but I don't understand the problem quite well enough (and I ran out of debugging juice for today :D)

[raoulvdberge]

My next step was investigating why the ThreadBoundApplicationListenerAdapter’s delegate was not set for my task threads.
My approach was the following: set a breakpoint in Thread constructor where it passes the parent’s thread local map.

In the succeeding case (with roughly 4 tests), ThreadBoundApplicationListenerAdapter#registerDelegate was called first, and after that, the task thread is created by ThreadPoolTaskExecutor->CustomizableThreadFactory#newThread. In this case the thread local is passed properly.

However, in the failing case, with more tests, the task threads are created earlier (simply due to covering more code that triggers events etc). At this point the ThreadBoundApplicationListenerAdapter’s delegate is not set yet, so nothing is passed to the task threads.
By the time the PublishedEventsParameterResolver is created, there are already task threads. Calling InheritableThreadLocal#set will not propagate the ApplicationEventListener to the task threads anymore (this only happens on creation of the thread).

There appears to be an ordering issue: the event that is published from the task thread will never reach the test context’s PublishedEvents, simply because the task threads got initialised too early, before the Scenario parameter is resolved.

This also clarifies the flakiness in the test failures, it depends on when Spring decides to spawn more task threads and in which thread the task runs.

[raoulvdberge]

I have installed this patch locally and it seems to do the trick on my test suite.
Always pass the delegate from the main thread to the (potentially already spawned) task threads:

Index: spring-modulith-test/src/main/java/org/springframework/modulith/test/PublishedEventsParameterResolver.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/spring-modulith-test/src/main/java/org/springframework/modulith/test/PublishedEventsParameterResolver.java b/spring-modulith-test/src/main/java/org/springframework/modulith/test/PublishedEventsParameterResolver.java
--- a/spring-modulith-test/src/main/java/org/springframework/modulith/test/PublishedEventsParameterResolver.java	(revision 5a73087b7a879eb29514210c2e14402b99458ff0)
+++ b/spring-modulith-test/src/main/java/org/springframework/modulith/test/PublishedEventsParameterResolver.java	(date 1767524149125)
@@ -26,6 +26,8 @@
 import org.springframework.context.ApplicationEvent;
 import org.springframework.context.ApplicationListener;
 import org.springframework.context.support.AbstractApplicationContext;
+import org.springframework.core.task.TaskDecorator;
+import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
 import org.springframework.test.context.junit.jupiter.SpringExtension;
 import org.springframework.util.Assert;
 
@@ -38,6 +40,7 @@
 
 	private final Function<ExtensionContext, ApplicationContext> lookup;
 	private @Nullable ThreadBoundApplicationListenerAdapter listener;
+	private @Nullable PropagateDelegateFromMainThreadTaskDecorator taskDecorator;
 
 	PublishedEventsParameterResolver() {
 		this(ctx -> SpringExtension.getApplicationContext(ctx));
@@ -80,6 +83,10 @@
 			listener.registerDelegate(publishedEvents);
 		}
 
+		if (taskDecorator != null) {
+			taskDecorator.listenerFromMainThread = publishedEvents;
+		}
+
 		return publishedEvents;
 	}
 
@@ -106,6 +113,9 @@
 
 					return adapter;
 				});
+
+		taskDecorator = new PropagateDelegateFromMainThreadTaskDecorator(listener);
+		context.getBean(ThreadPoolTaskExecutor.class).setTaskDecorator(taskDecorator);
 	}
 
 	/*
@@ -165,4 +175,31 @@
 			}
 		}
 	}
+
+	private static final class PropagateDelegateFromMainThreadTaskDecorator implements TaskDecorator {
+		private final ThreadBoundApplicationListenerAdapter adapter;
+		@Nullable
+		private ApplicationListener<ApplicationEvent> listenerFromMainThread;
+
+		PropagateDelegateFromMainThreadTaskDecorator(ThreadBoundApplicationListenerAdapter adapter) {
+			this.adapter = adapter;
+		}
+
+		@Override
+		public Runnable decorate(Runnable task) {
+			return () -> {
+				if (listenerFromMainThread == null) {
+					task.run();
+					return;
+				}
+				ApplicationListener<ApplicationEvent> previous = adapter.delegate.get();
+				try {
+					adapter.delegate.set(listenerFromMainThread);
+					task.run();
+				} finally {
+					adapter.delegate.set(previous);
+				}
+			};
+		}
+	}
 }

Test hardening for PublishedEventsIntegrationTest:

Index: spring-modulith-test/src/test/java/org/springframework/modulith/test/PublishedEventsIntegrationTests.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/spring-modulith-test/src/test/java/org/springframework/modulith/test/PublishedEventsIntegrationTests.java b/spring-modulith-test/src/test/java/org/springframework/modulith/test/PublishedEventsIntegrationTests.java
--- a/spring-modulith-test/src/test/java/org/springframework/modulith/test/PublishedEventsIntegrationTests.java	(revision 5a73087b7a879eb29514210c2e14402b99458ff0)
+++ b/spring-modulith-test/src/test/java/org/springframework/modulith/test/PublishedEventsIntegrationTests.java	(date 1767551196907)
@@ -24,11 +24,15 @@
 import lombok.RequiredArgsConstructor;
 
 import org.awaitility.Awaitility;
+import org.junit.jupiter.api.MethodOrderer;
+import org.junit.jupiter.api.Order;
 import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestMethodOrder;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.boot.test.context.SpringBootTest;
 import org.springframework.context.ApplicationEventPublisher;
+import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
 
 /**
  * Integration tests for {@link PublishedEvents}.
@@ -38,12 +42,25 @@
 @RequiredArgsConstructor
 @ExtendWith(PublishedEventsExtension.class)
 @SpringBootTest(classes = TestConfiguration.class)
+@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
 class PublishedEventsIntegrationTests {
 
 	@Autowired ApplicationEventPublisher publisher;
 	@Autowired AsyncEventListener listener;
+	@Autowired ThreadPoolTaskExecutor taskExecutor;
+
+	@Test // #1513
+	@Order(1)
+	void putSomeLoadSoTheTaskThreadsAreInitializedEarlyBeforeTheScenarioRuns() {
+		for (int i = 0; i < 100; ++i) {
+			taskExecutor.execute(() -> {
+				// fake task
+			});
+		}
+	}
 
 	@Test // #116
+	@Order(2)
 	void capturesEventsTriggeredByAsyncEventListeners(PublishedEvents events) {
 
 		assertThatNoException().isThrownBy(() -> {

[raoulvdberge]

I made a minimal reproduction case:

import jakarta.transaction.Transactional;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationEventPublisher;
import org.springframework.context.annotation.Import;
import org.springframework.modulith.events.ApplicationModuleListener;
import org.springframework.modulith.test.ApplicationModuleTest;
import org.springframework.modulith.test.Scenario;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.stereotype.Component;

@ApplicationModuleTest
@Import(ReproducerTest.MyEventHandler.class)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class ReproducerTest {
	@Autowired
	private ThreadPoolTaskExecutor taskExecutor;

	@Test
	@Order(1)
	void putSomeLoadSoTheTaskThreadsAreInitializedEarlyBeforeTheScenarioRuns() {
		for (int i = 0; i < 999; ++i) {
			final int ii = i;
			taskExecutor.execute(() -> System.out.println("Hello " + ii));
		}
	}

	@Test
	@Order(2)
	void reproducer(final Scenario scenario) {
		System.out.println("Reproducer is running now");
		scenario.publish(new EventA())
			.andWaitForEventOfType(EventB.class)
			.toArrive();
	}

	@Component
	@Transactional
	public static class MyEventHandler {
		private final ApplicationEventPublisher publisher;

		public MyEventHandler(final ApplicationEventPublisher publisher) {
			this.publisher = publisher;
		}

		@ApplicationModuleListener
		void handleEventA(final EventA evt) {
			publisher.publishEvent(new EventB());
		}
	}

	private record EventA() {
	}

	private record EventB() {
	}
}

org.awaitility.core.ConditionTimeoutException: Lambda expression in org.springframework.modulith.test.Scenario$When$EventResult expected the predicate to return <true> but it returned <false> for input of <[]> within 10 seconds.