Webflux /actuator/** endpoints can become unresponsive during high throughput, even when exposing a separate management port

[filpano]

During a high-throughput usage of one of our endpoints, we noticed that our metrics were failing to be scraped. The actual scenario then looks something like this:

After some deeper analysis, we found out that this is due to the fact that in Webflux, a high-throughput endpoint could potentially bottleneck the event loop and cause many other endpoints - including /actuator/**, to fail to be processed in time or at all. This is made worse if the Webflux call erroneously contains a blocking call (e.g. a JPA call) which is not offloaded to a bounded scheduler.

Interestingly, the first indicator that this was happening was an AbortedException:

AbortedException: Connection has been closed BEFORE send operation

r.n.c.AbortedException: Connection has been closed BEFORE send operation
	at r.n.c.AbortedException.beforeSend(AbortedException.java:59)
	Suppressed: r.c.p.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ Handler Actuator web endpoint 'prometheus' [DispatcherHandler]
Original Stack Trace:
		at r.n.c.AbortedException.beforeSend(AbortedException.java:59)
		at r.n.h.s.HttpServerOperations.then(HttpServerOperations.java:685)
		at r.n.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:750)
		at r.n.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:739)
		at r.n.NettyOutbound.then(NettyOutbound.java:358)
		at r.n.h.s.HttpServerOperations.send(HttpServerOperations.java:544)
		at o.s.h.s.r.ReactorServerHttpResponse.writeWithInternal(ReactorServerHttpResponse.java:96)
		at o.s.h.s.r.AbstractServerHttpResponse.lambda$writeWith$2(AbstractServerHttpResponse.java:183)
		at o.s.h.s.r.AbstractServerHttpResponse.doCommit(AbstractServerHttpResponse.java:259)
		at o.s.h.s.r.AbstractServerHttpResponse.lambda$writeWith$5(AbstractServerHttpResponse.java:180)
		at r.c.p.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
		at r.c.p.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
		at r.c.p.Operators$ScalarSubscription.request(Operators.java:2570)
		at r.c.p.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136)
		at r.c.p.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
		at r.c.p.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139)
		at r.c.p.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136)
		at r.c.p.MonoFlatMap$FlatMapInner.onSubscribe(MonoFlatMap.java:291)
		at r.c.p.FluxContextWrite$ContextWriteSubscriber.onSubscribe(FluxContextWrite.java:101)
		at r.c.p.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:152)
		at r.c.p.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
		at r.c.p.FluxContextWrite$ContextWriteSubscriber.onSubscribe(FluxContextWrite.java:101)
		at r.c.p.MonoJust.subscribe(MonoJust.java:55)
		at r.c.p.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
		at r.c.p.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
		at r.c.p.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
		at r.c.p.Operators$MonoInnerProducerBase.complete(Operators.java:2864)
		at r.c.p.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)
		at r.c.p.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152)
		at r.c.p.Operators$ScalarSubscription.request(Operators.java:2572)
		at r.c.p.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
		at r.c.p.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103)
		at r.c.p.Operators$MonoInnerProducerBase.request(Operators.java:2931)
		at r.c.p.Operators$MultiSubscriptionSubscriber.set(Operators.java:2366)
		at r.c.p.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2240)
		at r.c.p.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115)
		at r.c.p.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
		at r.c.p.FluxJust.subscribe(FluxJust.java:68)
		at r.c.p.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
		at r.c.p.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
		... 14 frames truncated

which it seems was Prometheus trying to call the /actuator/prometheus endpoint, hitting some timeout, and cancelling the request.

What made this even harder to debug was the following log message from the org.springframework.web.server.adapter.HttpWebHandlerAdapter:

[4728cf52-74336] Error [org.springframework.http.converter.HttpMessageNotWritableException: No Encoder for [com.acme.error.handling.ErrorResponse] with preset Content-Type 'application/openmetrics-text;version=1.0.0;charset=utf-8'] for HTTP GET "/actuator/prometheus", but ServerHttpResponse already committed (200 OK)

(note: com.acme.error.handling.ErrorResponse is our own GlobalExceptionHandler trying to wrap the response error)

which is understandable given the above exception, although the logged response code (200 OK) seems a bit iffy.

While debugging this, we found out that we used the same server.port as well as management.server.port. According to the docs, setting an explicit management.server.port different from server.port should start a separate Netty which manages its own threadpool to protect against such use cases. Indeed, setting:

server.port: 8080
management.server.port: 8081

seems to do so:

2025-07-24T13:47:58.622Z  INFO 1 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2025-07-24T13:47:58.660Z  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 14 endpoint(s) beneath base path '/actuator'
2025-07-24T13:47:58.682Z  INFO 1 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8081

However, addressing this Netty:

does not solve the problem. In fact, reducing the standard Reactor Threadpool sizes, using e.g. JAVA_TOOL_OPTIONS=-Dreactor.schedulers.defaultPoolSize=4 -Dreactor.schedulers.defaultBoundedElasticSize=4 -Dreactor.schedulers.defaultBoundedElasticQueueSize=10

makes it relatively easy to reproduce an error like the following:

Test

2025-07-24T13:48:28.293Z ERROR 1 --- [or-http-epoll-3] a.w.r.e.AbstractErrorWebExceptionHandler : [c6475471-269]  500 Server Error for HTTP GET "/actuator/prometheus"
2025-07-24T13:48:28.310912574Z 
2025-07-24T13:48:28.310918808Z reactor.core.Exceptions$ReactorRejectedExecutionException: Task capacity of bounded elastic scheduler reached while scheduling 1 tasks (11/10)
2025-07-24T13:48:28.310923097Z 	at reactor.core.Exceptions.failWithRejected(Exceptions.java:293) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310926836Z 	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
2025-07-24T13:48:28.310930136Z Error has been observed at the following site(s):
2025-07-24T13:48:28.310933355Z 	*__checkpoint ⇢ Handler org.springframework.boot.actuate.endpoint.web.reactive.AbstractWebFluxEndpointHandlerMapping$ReadOperationHandler#handle(ServerWebExchange) [DispatcherHandler]
2025-07-24T13:48:28.310938710Z 	*__checkpoint ⇢ org.springframework.web.filter.reactive.ServerHttpObservationFilter [DefaultWebFilterChain]
2025-07-24T13:48:28.310942459Z 	*__checkpoint ⇢ HTTP GET "/actuator/prometheus" [ExceptionHandlingWebHandler]
2025-07-24T13:48:28.310946021Z Original Stack Trace:
2025-07-24T13:48:28.310949246Z 		at reactor.core.Exceptions.failWithRejected(Exceptions.java:293) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310952578Z 		at reactor.core.scheduler.BoundedElasticScheduler$BoundedScheduledExecutorService.ensureQueueCapacity(BoundedElasticScheduler.java:922) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310967825Z 		at reactor.core.scheduler.BoundedElasticScheduler$BoundedScheduledExecutorService.submit(BoundedElasticScheduler.java:1022) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310971149Z 		at io.micrometer.core.instrument.internal.TimedScheduledExecutorService.submit(TimedScheduledExecutorService.java:85) ~[micrometer-core-1.11.0.jar!/:1.11.0]
2025-07-24T13:48:28.310973763Z 		at reactor.core.scheduler.DelegatingScheduledExecutorService.submit(DelegatingScheduledExecutorService.java:84) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310976424Z 		at reactor.core.scheduler.Schedulers.directSchedule(Schedulers.java:1244) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310978942Z 		at reactor.core.scheduler.BoundedElasticScheduler.schedule(BoundedElasticScheduler.java:304) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310981366Z 		at reactor.core.scheduler.Schedulers$CachedScheduler.schedule(Schedulers.java:1163) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310984892Z 		at reactor.core.publisher.MonoSubscribeOnCallable.subscribe(MonoSubscribeOnCallable.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310988045Z 		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310990569Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310993206Z 		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310995652Z 		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2545) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.310998141Z 		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311000808Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311003469Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311005829Z 		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2341) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311008374Z 		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311010813Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311013717Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311016306Z 		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311018988Z 		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311025846Z 		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311029219Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311032370Z 		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311034961Z 		at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311037318Z 		at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311039652Z 		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:292) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311042683Z 		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:187) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311045319Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311048008Z 		at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:293) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311050758Z 		at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:474) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311053379Z 		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311055908Z 		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2545) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311058402Z 		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311060865Z 		at reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:466) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311064703Z 		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:152) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311067310Z 		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311069869Z 		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311072226Z 		at reactor.core.publisher.MonoZip$ZipCoordinator.request(MonoZip.java:216) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311074567Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311077649Z 		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onSubscribe(MonoIgnoreThen.java:134) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311083407Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311086382Z 		at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:125) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311088862Z 		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311091365Z 		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311093732Z 		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311096230Z 		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311098592Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onComplete(MonoFlatMap.java:189) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311100958Z 		at reactor.core.publisher.Operators.complete(Operators.java:137) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311103397Z 		at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:121) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311105897Z 		at reactor.core.publisher.Mono.subscribe(Mono.java:4485) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311108353Z 		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311110738Z 		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311113103Z 		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311115534Z 		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311117887Z 		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311120466Z 		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311122960Z 		at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311125762Z 		at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:258) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311128424Z 		at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:863) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311130878Z 		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311133899Z 		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311139036Z 		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2545) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311141672Z 		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311144040Z 		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311146438Z 		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2305) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311148971Z 		at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:338) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311151631Z 		at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311154081Z 		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2341) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311156523Z 		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2215) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311158896Z 		at reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:70) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311161282Z 		at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onSubscribe(FluxConcatMapNoPrefetch.java:164) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311163833Z 		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311166542Z 		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311168922Z 		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311171420Z 		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311173894Z 		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311176263Z 		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311178550Z 		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311180915Z 		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311183351Z 		at reactor.core.publisher.Mono.subscribe(Mono.java:4485) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311185599Z 		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311190533Z 		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311193200Z 		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311195794Z 		at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55) ~[reactor-core-3.5.6.jar!/:3.5.6]
2025-07-24T13:48:28.311198204Z 		at reactor.netty.http.server.HttpServer$HttpServerHandle.onStateChange(HttpServer.java:1006) ~[reactor-netty-http-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311200611Z 		at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:710) ~[reactor-netty-core-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311203067Z 		at reactor.netty.transport.ServerTransport$ChildObserver.onStateChange(ServerTransport.java:481) ~[reactor-netty-core-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311205454Z 		at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:621) ~[reactor-netty-http-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311207829Z 		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113) ~[reactor-netty-core-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311210597Z 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311213117Z 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311215662Z 		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311218313Z 		at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:230) ~[reactor-netty-http-1.1.7.jar!/:1.1.7]
2025-07-24T13:48:28.311220637Z 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311223030Z 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311225555Z 		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311227973Z 		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311230456Z 		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311232889Z 		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311238413Z 		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311241410Z 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311243923Z 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311246769Z 		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311249541Z 		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311251964Z 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311254425Z 		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311256933Z 		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311259381Z 		at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[netty-transport-classes-epoll-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311267587Z 		at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) ~[netty-transport-classes-epoll-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311275056Z 		at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[netty-transport-classes-epoll-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311278051Z 		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311295568Z 		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311301775Z 		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.92.Final.jar!/:4.1.92.Final]
2025-07-24T13:48:28.311303333Z 		at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]

where the Netty running on port 8081 suffers a RejectedExecutionException while the Netty running on port 8080 is getting a lot of requests.

I haven't looked much more deeply than this at the moment, but this seems to imply that despite starting a separate Netty Server, the same Reactor schedulers and threadpools are shared (possibly the shared Schedulers.parallel() instance) as opposed to using e.g. Schedulers.newParallel(), which IMO defeats the purpose of setting the management.server.port. Somewhat obviously, scraping /actuator/prometheus content via JMX as opposed to HTTP would be the only sensible workaround that I can see at the moment. This is a rather large change for something that seems like it should already work, however.

I have created a small demo project which showcases this issue. See the included README.md and docker-compose.yaml.

webflux-issue.zip

[filpano]

Some further analysis (included in the demo):

1. Using the standard reactor threadpool (i.e. no subscribeOn(...) with a potentially blocking call (bad practice - just for testing purposes):
   -> Blocks /actuator/** calls
2. Using the standard reactor bounded elastic (using .subscribeOn(Schedulers.boundedElastic())):
   -> Blocks /actuator/** calls (somewhat surprisingly)
3. Using a custom bounded scheduler (using e.g. Schedulers.newBoundedElastic(4, 10, "customScheduler")):
   -> Does not block /actuator/** calls.

Just to clarify, there isn't anything special about the /actuator/** calls themselves - ordinary application endpoints would themselves also potentially be bullied out of a spot in the event loop by another, high-throughput endpoint.

The issue here, IMO, is that starting a separate Netty via management.server.port and scraping /actuator/** endpoints via that Netty does not seem to prevent this.

Using a custom scheduler for high-throughput endpoints does alleviate the issue, but is only a band-aid. At the very least, it should be possible to provide a custom scheduler to be used exclusively for /actuator** calls (or rather, for the Netty running on the management.server.port).

[bclozel]

Thanks for the detailed report @filpano ! I'm working on Spring Framework and Spring Boot, so hopefully I can bring an accurate perspective from both teams.

First, let me say that I'm sorry that you had to deal with such a situation. It's clearly not easy to pinpoint exactly the problem here.

Analysis

Looking at your first graph, it seems that this was produced with a sample project calling a blocking method in the middle of a reactive pipeline, on purpose, to demonstrate the problem. I assume that the production use case would look somewhat the same, except that the CPU usage should be much higher, probably around 80%? Either way, I agree that losing metrics is not ideal. As for the initial symptoms, I guess it is linked with network timeouts and connections being closed because they couldn't be processed in time. I'm not sure how we can improve the situation here, but that's something I'll think about.

According to the docs, setting an explicit management.server.port different from server.port should start a separate Netty which manages its own threadpool to protect against such use cases.

I'm not sure where this was explained in the docs, can you point me to the exact section please? I think we have something relevant in the Spring Boot docs, Kubernetes Probe section:

If your Actuator endpoints are deployed on a separate management context, the endpoints do not use the same web infrastructure (port, connection pools, framework components) as the main application. In this case, a probe check could be successful even if the main application does not work properly (for example, it cannot accept new connections). For this reason, it is a good idea to make the liveness and readiness health groups available on the main server port.

Arguably, we could improve the description here and state that in both cases (separate vs. same port) and for both Reactive/Servlet, CPU and memory resources will be shared: for WebFlux, the same event loop resources will be used and for Servlet (in virtual threads mode), the same forkjoin pool will be used.

I haven't looked much more deeply than this at the moment, but this seems to imply that despite starting a separate Netty Server, the same Reactor schedulers and threadpools are shared (possibly the shared Schedulers.parallel() instance) as opposed to using e.g. Schedulers.newParallel(), which IMO defeats the purpose of setting the management.server.port.

I'm not a Netty expert but I believe that by default request processing happens naturally on Netty's EventLoop with the configured resources in Reactor Netty. Non-blocking processing events are picked up by workers, each on its own thread. Usually resources are tailored by looking at the available resources (like CPU count on the server). You can choose to publish/subscribe work on specific Reactor Schedulers, but that's a different topic and this should be done with care.

In the Spring Boot case, starting a new Netty instance does not create a new set of resources. I think this is by design, as the goal here is to optimize as much as possible resource usage. For the same reason, resources are also shared with the WebClient at runtime.

Some further analysis (included in the demo):

I'm not really sure how to explain what you're seeing here. Maybe creating an issue on the reactor tracker would be helpful.
My intuition tells me that using Schedulers.boundedElastic() is the right one for scheduling blocking I/O operations that potentially take long. Overwhelming this solution takes a lot of queued tasks and a lot of concurrent work given the default configuration of this scheduler. On the other hand Schedulers.newBoundedElastic(4, 10, "customScheduler") is quite limited with only 4 threads and 10 tasks being queued!

Possible solutions

I have considered your proposal about separating resources for server instances in Spring Boot.
I think this would have major downsides:

• it is likely that each set of resources will step on the other. The main advantage of the netty event loop is to use a platform thread to its maximum on a particular CPU and avoid context switching at all costs (just like virtual threads). If not done 100% properly we will have competing threads for the same resources.
• it is very hard to decide how to size those resource pools
• if we manage to reserve 1 CPU out of 4 for management, your main application might be completely unresponsive while your prometheus metrics are showing CPU usage at 70%

Let's reconsider the problem under different angles. If the main issue is that the application and metrics are unresponsive when CPU usage is maxed out, I would say this is a well-known problem even for non-reactive systems. At 80% usage, things are not predictable anymore. On the other hand, if the main issue is about blocking work derailing the entire application, this is also unfortunately a well-known issue and the Reactor team built tools to detect that.

I am also wondering whether resources a properly tailored for the environment. It would be interesting to know for your production environment how many CPUs are allocated to your VM/container, how many the JVM is seeing and as a result what the Schedulers and EventLoop resources look like. That would at least explain why an under-sized bounded elastic scheduler would perform better than the default one.

I don't have concrete plans for improving this situation right now. We can certainly improve the reference docs in Spring Boot and other places (depending on where you've been looking for information).

Thanks!

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[filpano]

Thank you very much for the detailed feedback @bclozel !

Looking at your first graph, it seems that this was produced with a sample project calling a blocking method in the middle of a reactive pipeline, on purpose, to demonstrate the problem. I assume that the production use case would look somewhat the same, except that the CPU usage should be much higher, probably around 80%? Either way, I agree that losing metrics is not ideal. As for the initial symptoms, I guess it is linked with network timeouts and connections being closed because they couldn't be processed in time. I'm not sure how we can improve the situation here, but that's something I'll think about.

Yes, this graph was generated by the sample project I provided in the original post. The thing I wanted to note with the graph is that this behaviour doesn't seem driven by CPU usage by itself but rather the resource starvation that occurs inside the reactor threadpools. Indeed, the "blocking" calls in the provided sample are just calling Thread.sleep(1000) to simulate something taking up a slot for a long time, but not necessarily doing anything meaningful that would drive up CPU usage.

This is supported by the fact that I have setup a very short scrape interval (~2s) - if much work was being done I'd expect the first successful scrapeafter things calm down to show a CPU usage that is bit higher, though it's certainly difficult to tell for sure.

I think we have something relevant in the Spring Boot docs, Kubernetes Probe section:
[...]

Yes, this is the section I was referring to, in particular the "for example, it cannot accept new connections" section.

I'm not a Netty expert but I believe that by default request processing happens naturally on Netty's EventLoop with the configured resources in Reactor Netty. Non-blocking processing events are picked up by workers, each on its own thread. Usually resources are tailored by looking at the available resources (like CPU count on the server). You can choose to publish/subscribe work on specific Reactor Schedulers, but that's a different topic and this should be done with care.
In the Spring Boot case, starting a new Netty instance does not create a new set of resources. I think this is by design, as the goal here is to optimize as much as possible resource usage. For the same reason, resources are also shared with the WebClient at runtime.

I see. I hadn't dived deep into NIO's Event Loops, so I was under the mistaken assumption that Netty explicitly delegates to a Reactor Scheduler somewhere - now I see that this is not the case, which makes the issue a bit more nuanced.

What is further interesting is that, it seems like the default Schedulers setting (in particular - the default number of queued tasks, 100_000) are not very well suited to a high-throughput workload regardless of actual CPU usage under the following conditions:

1. More requests come in within a given time frame than can be processed
2. No explicit backpressure is set up (not sure if this is possible via Netty itself?)

This is just a common backpressure issue, but it's interesting what it led to. Our initial finding that something was amiss with our service was a failing Docker health check that looks as follows:

HEALTHCHECK --interval=10s --timeout=5s --retries=30 CMD nc -z localhost 8080 && exit 0 || exit 1

which is dead simple and uses netcat to check if a port is listening. This healtcheck was failing over a period of 5m (30*10s).

I don't have the exact numbers for our initial incident on hand, but we were seeing up to ~9000 (no pun intended) open TCP connections / sockets in use at any given moment inside the containers experiencing the issue. ulimit -n is unlimited, so it seems like we definitely hit a resource starvation somewhere in the application stack.

An example from the demo project using the ss -s command also shows interesting numbers after sending about ~3000 requests in a short amount of time (with minimal worker thread counts to exacerbate the issue):

$ ss -s
TCP:   9688 (estab 2414, closed 7271, orphaned 0, timewait 0)

Transport Total     IP        IPv6
RAW       0         0         0        
UDP       1         1         0        
TCP       2417      1         2416     
INET      2418      2         2416     
FRAG      0         0         0        

which shows that the "waiting" connections are queued and hold on to an open socket for the duration of the wait (makes sense).

This leads me to suspect that something like the following chain of events took place:

1. Relatively high number of requests come in in a given time frame (>25/s)
2. These aren't processed fast enough, so a lot end up being queued
3. Clients potentially have their own timeouts, leading them to retrying the request even though they were queued
4. Queue size keeps increasing due to no backpressure
5. Eventually, memory pressure from all the connection allocations lead to increased GC pauses -> higher CPU usage, even though no more "meaningful" work is done past the maximum number of concurrent processing allowed
6. Since the queue keeps growing, memory pressure keeps increasing until GC can't keep up; it's likely there is some GC thrashing going on at this point
7. More sockets are held by incoming connections that will end up exacerbating the above symptoms
8. At this point it's not unlikely that the kernel’s TCP backlog is full too

Unfortunately, due to having no metrics for the given problematic time frame, I can't back up points 5 and higher with concrete numbers other than the observed very high number of open TCP connections / sockets.

I'm not really sure how to explain what you're seeing here. Maybe creating an issue on the reactor tracker would be helpful. My intuition tells me that using Schedulers.boundedElastic() is the right one for scheduling blocking I/O operations that potentially take long. Overwhelming this solution takes a lot of queued tasks and a lot of concurrent work given the default configuration of this scheduler. On the other hand Schedulers.newBoundedElastic(4, 10, "customScheduler") is quite limited with only 4 threads and 10 tasks being queued!

I agree. The behaviour doesn't make a lot of sense to me, especially taking into account the other potential layer of resource starvation/contention above (i.e. I would have expected both boundeElastic schedulers to behave similarly). Raising an issue on the reactor tracker seems like the right call on this, thanks.

I have considered your proposal about separating resources for server instances in Spring Boot. I think this would have major downsides:

• it is likely that each set of resources will step on the other. The main advantage of the netty event loop is to use a platform thread to its maximum on a particular CPU and avoid context switching at all costs (just like virtual threads). If not done 100% properly we will have competing threads for the same resources.
• it is very hard to decide how to size those resource pools
• if we manage to reserve 1 CPU out of 4 for management, your main application might be completely unresponsive while your prometheus metrics are showing CPU usage at 70%

Let's reconsider the problem under different angles. If the main issue is that the application and metrics are unresponsive when CPU usage is maxed out, I would say this is a well-known problem even for non-reactive systems. At 80% usage, things are not predictable anymore. On the other hand, if the main issue is about blocking work derailing the entire application, this is also unfortunately a well-known issue and the Reactor team built tools to detect that.

Thanks for the clear summary. I did not know about the BlockHound tool - we'll definitely take a look.

I am also wondering whether resources a properly tailored for the environment. It would be interesting to know for your production environment how many CPUs are allocated to your VM/container, how many the JVM is seeing and as a result what the Schedulers and EventLoop resources look like. That would at least explain why an under-sized bounded elastic scheduler would perform better than the default one.

That's a good point. The containers that encountered this issue are not very powerful:

• 1GB memory
• 1 vCPU
• -Xmx512m

so the default threadpool configurations are definitely on the limited side. What's interesting is that I've run the provided Demo project on different machines with e.g. 14 cores and no heap limitation and still observed the same behaviour (it's possible that the route it took to get there was different, but the end result was what I described above).

---

To sum up, I can definitely appreciate that this isn't a trivial problem to solve, and many of the default configurations have been working fine for us thus far. We might simply have gone into "needs application specific tuning" territory, and we'll do more stress testing of these components to figure out what we can do.

If any of this additional information seems like it might indicate some deeper underlying issue that we could take a better look at, that would be great. Otherwise I'll try following your suggestions and raise an issue on the Reactor Tracker re.: boundedElastic() and try out BlockHound.

Thank you!

[mateusz-nalepa]

Hey @filpano @bclozel
Due to curiosity, I was just scrolling through github issues, cause recently I've opened #35302 and for me it seems, that together with @filpano we've encountered exactly same problem: Thread Pool Saturation

Looking at your scenarios:

1. Using the standard reactor threadpool (i.e. no subscribeOn(...) with a potentially blocking call (bad practice - just for testing purposes):
   -> Blocks /actuator/** calls
2. Using the standard reactor bounded elastic (using .subscribeOn(Schedulers.boundedElastic())):
   -> Blocks /actuator/** calls (somewhat surprisingly)
3. Using a custom bounded scheduler (using e.g. Schedulers.newBoundedElastic(4, 10, "customScheduler")):
   -> Does not block /actuator/** calls.

Scenario 1:

• requests in UserController are executed on reactor Event Loop threads
• those requests are blocking
• that's why endpoints /actuator/** or any other endpoints are unresponsive

Scenario 2:

• Put a debug in PrometheusScrapeEndpoint#scrape, and you will see, that this endpoint is being executed on boundedElastic()
• please check also AbstractWebFluxEndpointHandlerMapping.ElasticSchedulerInvoker#invoke for this scenario
• Schedulers.boundedElastic() is cached, so you can use it wherever you want and thread pool will be exactly the same
• default threads number for boundedElastic() is 10 * Runtime.getRuntime().availableProcessors()
• for reference please check Schedulers#BOUNDED_ELASTIC_SUPPLIER
• when you started executing Thread.sleep() 1000 times, then as you've wrote These aren't processed fast enough, so a lot end up being queued
• so endpoints /actuator/** are unresponsive, but any other endpoints should be fine

Scenario 3:

• different thread pool used and voilà
• cpu does do a context-switch between boundedElastic() and newBoundedElastic(4, 10, "customScheduler")
• tasks from one thread pool does not affect tasks in another thread pool

So basically there are additional context-switches, but actuator endpoint or any other endpoint are responsive 🚀

Please let me know @filpano if this solves your problem

[bclozel]

@filpano your latest comment makes me think that the root problem is a blocking within a reactive pipeline. The reactive runtime works with a limited number of threads and the assumption that all I/O calls are asynchronous and will not block the current thread. Typically, calling Thread.sleep() within a reactive pipeline simulates well what could very much be happening in production for you: performing blocking I/O, holding on worker threads and tasks queuing.

Even if the server characteristics are quite light (the JVM itself will take a significant amount of resources for GC and JIT compiling), what you're describing fits very well with the case above. I think testing with BlockHound is very likely to detect blocking calls.

@mateusz-nalepa same thing here, if you are performing blocking operations and those are not scheduled on a boundedElastic, things will fail. Now Spring Boot schedules some non-reactive endpoints on a bounded elastic scheduler on purpose. But we're not magically overcoming all system limitations: if the thread pool is overwhelmed (here, because there are a lot of tasks and the initial thread poll size is small, because there is a single CPU available), I'm not sure what kind of strategy we can adopt here.

Let's wait for @filpano feedback with BlockHound.