ee8 AsyncResponse.resume() from non-Jetty thread causes IllegalStateException in HttpChannelState.unhandle()

[OmprakashD20]

Jetty version(s)
Broken: 12.1.6, 12.1.7
Working: 12.1.1 through 12.1.5

Jetty Environment
ee8

HTTP version
HTTP/1.1

Java version/vendor
OpenJDK 17

OS type/version
Linux

Description

After upgrading from 12.1.5 to 12.1.6, Jersey JAX-RS async responses resumed from a non-Jetty thread throw IllegalStateException in HttpChannelState.unhandle(). The response never reaches the client, resulting in a gateway timeout (504).

The pattern: a ScheduledThreadPoolExecutor thread calls AsyncResponse.resume(). Jersey writes the response and calls AsyncContext.complete(), which triggers HttpChannelState.complete() → runInContext() → HttpChannel.handle() → HttpChannelState.unhandle(). The unhandle() finds _state=IDLE instead of HANDLING.

Stack trace:

org.glassfish.jersey.server.ServerRuntime$Responder writeResponse
SEVERE: Error while closing the output stream in order to commit response.
java.lang.IllegalStateException: s=IDLE rs=COMPLETED os=COMPLETED is=IDLE awp=false se=false i=false al=0
at org.eclipse.jetty.ee8.nested.HttpChannelState.unhandle(HttpChannelState.java:409)
at org.eclipse.jetty.ee8.nested.HttpChannel.handle(HttpChannel.java:567)
at org.eclipse.jetty.ee8.nested.HttpChannel.run(HttpChannel.java:438)
at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1697)
at org.eclipse.jetty.ee8.nested.ContextHandler.handle(ContextHandler.java:941)
at org.eclipse.jetty.ee8.nested.HttpChannelState.runInContext(HttpChannelState.java:1050)
at org.eclipse.jetty.ee8.nested.HttpChannelState.complete(HttpChannelState.java:655)
at org.eclipse.jetty.ee8.nested.AsyncContextState.complete(AsyncContextState.java:60)
at org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl.complete(AsyncContextDelegateProviderImpl.java:102)
at org.glassfish.jersey.servlet.internal.ResponseWriter.commit(ResponseWriter.java:174)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:404)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:721)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:370)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:871)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:903)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:859)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

Analysis

Bisected to 12.1.6. Works on 12.1.5, broken on 12.1.6 and 12.1.7.

The regression was introduced by #14370 (fix for #14327), which changed how the core HttpChannelState handles failure/abort during response writing.

The core HttpChannelState changes affect the ee8 async complete path: when AsyncResponse.resume() is called from a non-Jetty thread, the core layer now completes/aborts the response in a way that leaves the ee8 HttpChannelState in s=IDLE rs=COMPLETED os=COMPLETED before unhandle() is called, causing the IllegalStateException.

The ee8-nested sources (HttpChannelState.java, HttpChannel.java, HttpOutput.java) are unchanged between 12.1.5 and 12.1.6 — only logging format changes.

How to reproduce?

1. Use Jetty ee8 with Jersey (@Suspended AsyncResponse)
2. Suspend the request, process work on a ScheduledThreadPoolExecutor
3. Call asyncResponse.resume(response) from the executor thread
4. The response commit fails with IllegalStateException, client never receives the response

Works on 12.1.5, fails on 12.1.6+.

[joakime]

Bisected to 12.1.6. Works on 12.1.5, broken on 12.1.6 and 12.1.7.

Since you are pointing out issues with 12.1.6 and 12.1.7.
The problem might lie somewhere else, namely logging.

If you are running your project via maven, and the jetty-ee8-maven-plugin, make sure you use <deployMode>FORK</deployMode> in your configuration. (This will work around a classloader bug in Maven 3.x with their old slf4j-api usage that breaks things in 12.1.6 and 12.1.7) An alternative would be to use maven 4.x (which has this fixed).

If you are running on an embedded scenario, make sure you are using slf4j-api 2.x (not slf4j-api 1.7.x or older).

[OmprakashD20]

@joakime

If you are running your project via maven, and the jetty-ee8-maven-plugin, make sure you use FORK in your configuration. (This will work around a classloader bug in Maven 3.x with their old slf4j-api usage that breaks things in 12.1.6 and 12.1.7) An alternative would be to use maven 4.x (which has this fixed).

I am using gradle to run my project and not maven. I am not using jetty-ee8-maven-plugin

If you are running on an embedded scenario, make sure you are using slf4j-api 2.x (not slf4j-api 1.7.x or older).

Yeah, I am running as an embedded server. How does slf4j-api could affect this?

[lorban]

@OmprakashD20 I tried reproducing you problem against the following Jersey service:

    @PUT
    @Path("/string")
    public void putString(@Context HttpServletRequest httpRequest, String data, @Suspended AsyncResponse asyncResponse)
    {
        Timer t = new Timer(true);
        t.schedule(new TimerTask() {
            @Override
            public void run()
            {
                asyncResponse.resume("Got " + data);
            }
        }, 1000L);
    }

Everything worked as expected, so I assume you're doing things slightly differently.

Could you please provide us with a reproducer or give us more details about what exactly the service that gives you trouble does?