Wait for pending body writes before completing Netty responses

[rogierslag]

Why

We encountered this through clients occasionally receiving 200 OK responses whose JSON ended prematurely. The captured bodies repeatedly stopped at exact multiples of 1,000 bytes, matching ChunkOutputStream’s chunk size. Corresponding server errors showed DefaultHttpContent reaching Netty’s encoder in state: 0, after response termination. A deterministic reproducer then confirmed that LastHttpContent could overtake a pending body write.

2026-08-17T14:54:21.041Z ERROR [rest-api]
logger=me.magnet.ResponseWriteFailureHandler
event_name=netty_response_write_failed
failure_stage=async_write
failed_write_is_last=false
cause_type=io.netty.handler.codec.EncoderException

io.netty.handler.codec.EncoderException:
java.lang.IllegalStateException:
unexpected message type: DefaultHttpContent, state: 0
    at io.netty.handler.codec.http.HttpObjectEncoder.write(...)
    at me.magnet.ResponseWriteFailureHandler.write(...)
    at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(...)
Caused by: java.lang.IllegalStateException:
unexpected message type: DefaultHttpContent, state: 0
    at io.netty.handler.codec.http.HttpObjectEncoder.encodeNotHttpMessageContentTypes(...)
    at io.netty.handler.codec.http.HttpObjectEncoder.encode(...)

Summary

RESTEasy’s Netty adapter can currently emit LastHttpContent while an earlier response-body write is still pending in the Netty pipeline.

When that happens, the client receives a valid HTTP response boundary after only part of the body. A late body chunk is rejected by Netty with:

EncoderException:
IllegalStateException:
unexpected message type: DefaultHttpContent, state: 0

For JSON responses, this results in an apparently successful 200 OK containing JSON truncated at the adapter’s 1,000-byte chunk boundary.

This PR makes response completion follow the actual asynchronous writes:

• every body write is tracked through its Netty promise;
• LastHttpContent is written only after all body writes succeed;
• body or terminal-write failure closes the connection instead of cleanly terminating a partial response;
• response termination is exactly once;
• sendError() is treated as a terminal full response and its future is reused by any subsequent finish();
• HEAD, empty, keep-alive and non-keep-alive responses use the same completion model.

The change does not block the Netty event loop, introduce another body copy or add an unbounded collection. It adds constant state per response and one completion listener with a short synchronized callback per emitted body chunk. It does not introduce backpressure.

The regression tests reproduce the production failure with plain Jackson JSON:

Before:
HttpResponse → 1,000-byte HttpContent → LastHttpContent → remaining HttpContent
                                                       ↳ EncoderException, state: 0

After:
HttpResponse → 1,000-byte HttpContent → remaining HttpContent → LastHttpContent

Thanks for maintaining this integration. We encountered this in production and put together a deterministic reproducer and a proposed fix. I’d particularly appreciate your view on coordinating completion in ChunkOutputStream, and whether you’d prefer the related sendError() changes in a separate PR.

Detailed failure analysis and implementation notes

What happens

ChunkOutputStream splits response bodies into chunks of 1,000 bytes and sends each chunk using ctx.writeAndFlush(...).

The old response lifecycle effectively was:

Jackson writes JSON to ChunkOutputStream
    |
    +-- DefaultHttpContent: bytes 0–999
    +-- DefaultHttpContent: bytes 1000–1999
    +-- DefaultHttpContent: remaining bytes
    |
Jackson returns
    |
NettyHttpResponse.finish()
    |
    +-- flush OutputStream
    +-- write LastHttpContent

The individual Netty writes can still be pending when finish() runs. If one of them is delayed, the actual outbound sequence becomes:

HttpResponse
DefaultHttpContent: bytes 0–999
LastHttpContent
DefaultHttpContent: remaining bytes

LastHttpContent is not merely another message. For a chunked HTTP/1.1 response it produces the final zero-sized chunk. According to RFC 9112 section 7.1, receiving that chunk completes the chunked transfer.

Hence the client is entitled to treat the response as finished after the first 1,000 bytes. It has no transport-level reason to reject the response. It only discovers the problem when the JSON parser encounters the incomplete document.

The late body chunk then reaches Netty’s encoder after termination. Netty correctly rejects it with:

io.netty.handler.codec.EncoderException:
java.lang.IllegalStateException:
unexpected message type: DefaultHttpContent, state: 0

In Netty, state: 0 is ST_INIT. Encoding LastHttpContent returns the encoder to that initial state. A subsequent DefaultHttpContent is therefore rejected as an unexpected message type.

This matches the production evidence:

• captured JSON prefixes end at exact multiples of 1,000 bytes;
• most prefixes end inside a JSON string or field name;
• other prefixes end between object or array entries;
• the server-side failure is a non-terminal DefaultHttpContent reaching the encoder in state: 0;
• the problem occurs without requiring GZIP compression.

Why the individual components still conform to their contracts

This failure is a bit counter-intuitive because none of the individual components has to behave incorrectly for it to happen.

A Jakarta REST MessageBodyWriter writes the entity to the provided OutputStream. It must not close that stream, because the container owns the response lifecycle. Jackson therefore returns once it has written its serialized representation to the stream.

The Java OutputStream.flush() contract requires buffered bytes to be passed to the underlying destination. It does not guarantee that the final recipient has received them. Even for an operating-system-backed stream, flushing only guarantees that the bytes have been handed to the operating system.

Netty makes that distinction explicit: all Netty I/O operations are asynchronous. A write call can return immediately while the operation is still pending. The returned ChannelFuture reports completion or failure.

Finally, the HTTP encoder is correct to reject content after LastHttpContent. Accepting that content would violate the HTTP response boundary and could mix bytes with a subsequent keep-alive response.

Simply stated:

• Jackson correctly finishes writing to the provided stream.
• The stream correctly hands the bytes to Netty.
• Netty correctly reports completion asynchronously.
• The HTTP encoder correctly treats LastHttpContent as final.
• The client correctly treats the zero-sized chunk as successful response completion.

The missing piece is in the bridge between these contracts. The adapter treats “all writes have been submitted” as though it means “all writes have completed”. It then emits the terminal marker without observing the futures which Netty provides for exactly this purpose.

What this changes

ChunkOutputStream now owns a response-wide write lifecycle.

For every emitted body chunk it:

1. increments a pending-write counter before calling Netty;
2. attaches a completion listener to that chunk’s promise;
3. records the first write failure;
4. decrements the counter when the promise completes.

finish() now:

1. flushes the current response output stream, including any wrapping interceptor stream;
2. marks the entity-output lifecycle as finished;
3. creates a response-wide promise;
4. writes response termination only when every registered body write has completed.

The resulting invariant is:

LastHttpContent is emitted exactly once, and only after every body write registered before response completion has succeeded.

If a body write fails after the response has been committed, the adapter closes the channel and fails the response-wide promise. Once some body bytes may have reached the client, closing the transport is the only safe result. Writing LastHttpContent would turn an incomplete body into an apparently successful response.

The response retains the root ChunkOutputStream separately from any interceptor-provided wrapper. This lets finish() flush the outer stream while still coordinating the actual Netty body writes at the root.

Response termination is also stored as a single future. Repeated completion attempts, full error responses, HEAD responses and normal response completion therefore share the same terminal operation rather than emitting competing terminal messages.

Why sendError() is part of the lifecycle change

sendError() writes a DefaultFullHttpResponse. A full response contains its complete entity and response boundary in one message. It is therefore already terminal; finish() must not append another empty response or LastHttpContent.

The old implementation did not model that:

sendError()
    |
    +-- writeAndFlush(DefaultFullHttpResponse)
    +-- response remains uncommitted
    +-- write future is discarded

finish()
    |
    +-- response appears uncommitted
    +-- write another empty FullHttpResponse

That is a separate pre-existing failure path, but it becomes part of this change because normal responses and error responses now share one response-wide completion model. Leaving sendError() outside that model would mean terminationFuture does not represent every way in which a response can terminate.

sendError() now performs one atomic terminal-state transition:

1. check that the response has not already been committed;
2. build and transform the full error response;
3. mark the response committed;
4. write the full response;
5. store that write’s future as terminationFuture.

Both sendError() and writeResponseTermination() are synchronized. As such, they cannot both observe an unterminated response and emit competing terminal messages.

If finish() runs after sendError(), ChunkOutputStream.finish() still completes its local lifecycle. However, writeResponseTermination() returns the existing error-response future instead of writing anything else.

The resulting sequence is:

sendError()
    |
    +-- committed = true
    +-- terminationFuture = write FullHttpResponse

finish()
    |
    +-- flush any remaining local stream state
    +-- reuse terminationFuture
    +-- do not write another response

This also preserves connection semantics. For non-keep-alive requests, the close listener follows the response-wide promise, which in turn follows the full error response’s write future. The connection therefore closes after that terminal write completes.

For keep-alive requests, a successfully written error response leaves the connection in a valid state for the next request. If the terminal write fails, the failure path closes the connection rather than leaving its response boundary uncertain.

Empty and HEAD responses

Responses without a body still need a terminal write, but have no pending chunk promises to wait for.

writeResponseTermination() therefore keeps the existing distinction:

• if the response was committed through the chunk stream, write LastHttpContent;
• otherwise, write an empty DefaultFullHttpResponse.

HEAD responses do not have a ChunkOutputStream, hence they call writeResponseTermination() directly. They still benefit from the shared terminationFuture: repeated completion attempts return the same future and do not emit another response.

Concurrency and ordering

Lifecycle state is guarded by the existing writeLock. This creates one ordering boundary for body submission, body completion and response termination.

There are a few relevant interleavings:

1. All body writes complete before finish()

   finish() observes zero pending writes and emits response termination immediately.

2. finish() runs while body writes are pending

   finish() records that termination was requested but does not write LastHttpContent. The listener completing the final body write emits termination.

3. A body write fails while finish() is waiting

   The failure is recorded and the channel is closed. Once the pending writes have settled, the response-wide promise completes exceptionally. No successful terminal content is written.

4. A body write completes concurrently with finish()

   Both paths inspect and update the counter while holding the same lock. Either finish() sees zero pending writes, or the final listener sees that finishing was requested. responseWriteStarted provides a final exactly-once guard around the terminal transition.

5. finish() is called more than once

   Later calls return the existing response-wide promise. They do not write another LastHttpContent.

6. sendError() races with normal termination

   sendError() and writeResponseTermination() synchronize on the response. Only one of them can create the terminal future. The other reuses it or observes that the response is already committed.

7. A writer attempts to write after finishing

   The write is rejected as committed. This is outside the normal Jakarta REST writer lifecycle: synchronous writers have returned, while asynchronous writers must only complete their returned stage after their writes have finished.

The terminal transition rechecks finishRequested, pendingWrites and responseWriteStarted. Some of these checks overlap with current call-site guarantees, but keeping them local makes completeResponse() enforce its own preconditions. A future call site cannot accidentally terminate the response while writes remain pending.

There is no blocking wait in any of these paths. In particular, the Netty event loop is not blocked while waiting for body promises. Completion continues through listeners.

Failure handling

A successful response now requires two things:

1. every registered body promise succeeds;
2. the terminal write succeeds.

The response-wide promise completes only after both conditions hold.

If a body write fails:

• the first failure is preserved as the response failure;
• the channel is closed immediately;
• remaining pending writes are allowed to settle;
• no terminal response marker is emitted;
• the response-wide promise completes exceptionally.

The first failure is retained because later failures may merely be consequences of closing the channel. If Netty reports a failed future without a cause, the adapter creates an IOException rather than attempting to fail a promise with null.

If response termination itself fails, the adapter also closes the channel and fails the response-wide promise. A failed terminal write means the client cannot safely determine the response boundary, particularly on a keep-alive connection.

trySuccess() and tryFailure() are used when completing the response-wide promise. The state machine is intended to complete it once, but these calls prevent an unexpected duplicate callback from replacing the original transport result with a secondary promise-completion exception.

Memory, allocation and garbage collection

This adds some bookkeeping, although it remains bounded.

Per response, ChunkOutputStream now retains:

• one pending-write counter;
• lifecycle booleans;
• the first write failure, if one occurs;
• one response-wide promise.

NettyHttpResponse retains:

• the root ChunkOutputStream;
• one terminal future once a terminal response has been written.

There is no collection of every promise or body chunk. Lifecycle memory is therefore constant per response rather than proportional to response size.

Each emitted DefaultHttpContent receives one additional completion listener. Because this is a bound method reference, it may result in one small short-lived allocation per emitted body chunk. The response-wide promise adds another short-lived allocation per response.

These objects remain reachable until their corresponding writes complete. Under normal circumstances they should die young and be handled by the young generation. Large responses already create one Netty promise and one copied ByteBuf per emitted chunk; this change adds lifecycle bookkeeping to that existing per-chunk work but does not add another body copy.

No new unbounded queue, buffer or retry mechanism is introduced.

If a downstream handler accepts a write but never completes its promise, the response will now remain pending instead of being terminated early. That can retain the response state and connection until the promise fails or the connection is closed. This is intentional fail-closed behaviour, but it means channel and request timeouts remain important. A handler which permanently loses a promise already violates Netty’s outbound contract.

CPU, throughput and latency

The hot-path cost is constant per emitted body chunk:

• one counter increment;
• one listener registration;
• one listener callback;
• one counter decrement;
• a short synchronized section in the callback.

The response body is still submitted using the same number of 1,000-byte chunks. There is no additional encoding, compression, copying, retry, polling or scanning. Complexity remains O(number of chunks).

The existing write methods already use writeLock to protect the shared chunk buffer. This change makes promise completion listeners use the same lock, so the writer thread and Netty event loop can briefly contend on response lifecycle state. The critical sections only update scalar fields and do not wait for network I/O.

sendError() also gains synchronization. This is not expected to be a hot path, and the lock covers one local state transition and one asynchronous write submission. It does not wait for that write to complete.

Normal response latency should remain effectively unchanged when body promises complete promptly. The terminal write follows as soon as the final body promise succeeds.

When the outbound pipeline is delayed, response completion is now delayed by the same amount. That is the intended behavioural change: previously the adapter reported a completed response while part of its body was still pending.

This change does not introduce backpressure. A writer can still enqueue body chunks faster than the transport sends them, and queued ByteBuf memory remains governed by Netty’s existing channel and writability behaviour. Adding bounded response backpressure would be a separate change.

Throughput could be affected for very large responses because the adapter currently emits relatively small 1,000-byte chunks, hence listener and synchronization overhead occurs frequently. No benchmark has been run, so I would not call this zero-cost. However, the change adds constant bookkeeping to operations for which Netty already creates promises and buffers. Changing the chunk size or introducing another streaming strategy would have broader compatibility and memory implications and is intentionally outside this fix.

Keep-alive correctness also matters for throughput. An invalid terminal marker can make a connection appear reusable while a previous response still has pending content. Waiting for the actual body lifecycle prevents those bytes from crossing the boundary into the next response.

How this is reproduced

The tests cover the lifecycle at two levels.

The server-level Jackson test delays outbound content after RESTEasy dispatch has returned. Without the fix, LastHttpContent reaches the encoder before the delayed body. With the fix, the client receives the complete response.

A failure-path test rejects a body write and verifies that:

• the connection closes;
• the original body-write failure is preserved;
• no successful LastHttpContent is emitted.

Two additional pipeline-level tests reproduce the shapes observed in production without GZIP:

1. A plain Jackson document is split into a 1,000-byte prefix ending inside a JSON string and a delayed 512-byte tail.
2. A plain Jackson document is split into a 1,000-byte prefix ending exactly between object entries and a delayed 13-byte tail.

Against the old implementation, both produce:

HttpResponse
HttpContent: 1,000 bytes
LastHttpContent
HttpContent: remaining bytes

The final write then fails with the same encoder exception:

EncoderException:
IllegalStateException:
unexpected message type: DefaultHttpContent, state: 0

With this change, both produce:

HttpResponse
HttpContent: 1,000 bytes
HttpContent: remaining bytes
LastHttpContent

The tests exercise both the repository’s Netty 4.1 line and Netty 4.2.16. The failure is therefore not specific to Netty 4.2; that version merely reports the invalid sequence clearly.

Scope

This does not change the public API, response wire format, chunk size, Jackson configuration or compression behaviour.

It does change when a response is considered complete:

• normal responses complete after all body and terminal writes succeed;
• full error responses complete when their retained terminal write succeeds;
• failed body or terminal writes close the connection;
• repeated termination returns the existing future rather than writing another terminal message.

The change does not attempt to recover a response after a body write has failed. Once a partial response may have reached the client, recovery is unsafe.

Effectively, the adapter now treats every terminal response path consistently: complete body or full response first, terminal boundary once, and transport failure instead of clean truncation.

[jamezp]

Thank you @rogierslag. This change looks good to me and thank you for the detailed notes. Could you squash the commit into a single commit?

[rogierslag]

Not a problem at all, happy to help!

[rogierslag]

As we are experiencing this problem in production, is there an indication when a release might be made available?