Some valid POST requests with no body are rejected in 9.1

[swallez]

We had users report that their application using the Java client was working fine with ES 9.0, but that some API calls were failing with 9.1. After some investigation, @l-trotta and found it was caused by PR #129302, released in 9.1.

This PR changed how Request.hasContent() is implemented. This is causing issues with some POST requests with no body and no Content-Type header, in particular http/2 requests relayed by the Cloud proxy (and converted to http/1). The ES server will assume these requests have a content while they actually do not.

If these requests do not have a Content-Type header, API version compatibility will fail and the request will be rejected.

Steps to reproduce

Create a 9.0.0 cluster on Elastic Cloud, and run this command (adding -v will show that http/2 is used):

curl -X POST \
  -H "Authorization: ApiKey <redacted> \
  -H "Accept: application/vnd.elasticsearch+json; compatible-with=9" \
  "https://<cluster-id>.elastic-cloud.com/_count"

{"count":103,"_shards":{"total":18,"successful":18,"skipped":0,"failed":0}}

Run that same command on a 9.1.0 cluster. It fails, indicating that a Content-Type=null is not compatible with the Accept header value:

{"error":{"root_cause":[{"type":"media_type_header_exception","reason":"Invalid media-type value on headers [Accept, Content-Type]"}],"type":"media_type_header_exception","reason":"Invalid media-type value on headers [Accept, Content-Type]","caused_by":{"type":"status_exception","reason":"A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. Accept=application/vnd.elasticsearch+json; compatible-with=9, Content-Type=null"}},"status":400}

Adding ?error_trace=true shows the root exception:

Caused by: org.elasticsearch.ElasticsearchStatusException: A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. Accept=application/vnd.elasticsearch+json; compatible-with=9, Content-Type=null
    at org.elasticsearch.server@9.1.2/org.elasticsearch.rest.RestCompatibleVersionHelper.getCompatibleVersion(RestCompatibleVersionHelper.java:76)

This exception means that contentTypeHeader is null as expected, but that hasContent is true which isn't expected. This probably comes from HttpBody.isEmpty() where the body is likely set, and to a HttpBody.Stream instance.

Workarounds

There are some workarounds, although they can be considered counter-intuitive:

• Force http/1: add --http1.1 to the curl command. Most modern http clients will automatically negotiate and upgrade to http/2 when available. Works with Elastic Cloud, but may not be effective in all environments.
• Add a Content-Type header: add -H 'Content-Type: application/vnd.elasticsearch+json; compatible-with=9 to the curl command. Many users and client libraries will not add a Content-Type to an empty request.

Note that setting Content-Length: 0 doesn't solve the issue.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

http/2 requests relayed by the Cloud proxy (and converted to http/1)

I think there's a problem here, the request from the client has no content (whether HTTP/1.1 or HTTP/2) but the request that Elasticsearch sees apparently does. These two things are semantically different in HTTP, but it looks like the proxy is converting "no content" to "empty content". I suspect the proxy might be adding a spurious Transfer-encoding: chunked header to the proxied request, which indicates that it does have content (even if it turns out to be zero bytes long).

In compatibility mode a request with content, whether empty or otherwise, requires a Content-Type header to go alongside its Accept header.

Many users and client libraries will not add a Content-Type to an empty request.

Subtle but important distinction: libraries should not add a Content-type header to a request with no content, because this would change it into a request with (empty) content. But libraries should specify the Content-type of a request with content, even if empty.

See for instance RFC 9110 § 8.6 paragraph 5:

A user agent SHOULD send Content-Length in a request when the method defines a meaning for enclosed content and it is not sending Transfer-Encoding. For example, a user agent normally sends Content-Length in a POST request even when the value is 0 (indicating empty content). A user agent SHOULD NOT send a Content-Length header field when the request message does not contain content and the method semantics do not anticipate such data.

[mhl-b]

I agree with David's comment. It does look like proxy sets Transfer-encoding: chunked on request with no content, knowingly. It is unnecessary and complicates downstream. Transfer-encoding: chunked means content size is unknown, can be 0 or more. Since it can be more than 0 we expect Content-Type. I believe we need to address this in proxy.

Linked PR #129302 addresses uncontrolled memory usage by buffering content, by deferring content processing to the RestHandler's world. That means we can tell if there is content either by looking at Content-Length or Transfer-Encoding.

[DaveCTurner]

These two things are semantically different in HTTP, but it looks like the proxy is converting "no content" to "empty content".

More precisely, in HTTP/2 a request with no content comprises just a HEADERS frame with both the END_HEADERS and END_STREAM bits set, and that's what we see on the wire:

HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 1, Length 148, POST /_search
        Length: 148
        Type: HEADERS (1)
        Flags: 0x05, End Headers, End Stream
            00.0 ..0. = Unused: 0x00
            ..0. .... = Priority: False
            .... 0... = Padded: False
            .... .1.. = End Headers: True
            .... ...1 = End Stream: True
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        [Pad Length: 0]
        Header Block Fragment […]: <<REDACTED>>
        [Header Length: 305]
        [Header Count: 7]
        Header: :method: POST
        Header: :scheme: https
        Header: :authority: <<REDACTED>>.eu-west-1.aws.found.io:9243
        Header: :path: /_search
        Header: authorization: Basic <<REDACTED>>
        Header: user-agent: curl/8.12.1
        Header: accept: application/vnd.elasticsearch+json; compatible-with=9
        [Full request URI: https://<<REDACTED>>.eu-west-1.aws.found.io:9243/_search]
        [Response in frame: 19]

In contrast, a request with empty content has a HEADERS frame with END_HEADERS set, followed by an empty DATA frame with the END_STREAM bit:

HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 1, Length 149, POST /_search
        Length: 149
        Type: HEADERS (1)
        Flags: 0x04, End Headers
            00.0 ..0. = Unused: 0x00
            ..0. .... = Priority: False
            .... 0... = Padded: False
            .... .1.. = End Headers: True
            .... ...0 = End Stream: False
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        [Pad Length: 0]
        Header Block Fragment […]: <<REDACTED>>
        [Header Length: 305]
        [Header Count: 7]
        Header: :method: POST
        Header: :scheme: https
        Header: :authority: <<REDACTED>>.eu-west-1.aws.found.io:9243
        Header: :path: /_search
        Header: authorization: Basic <<REDACTED>>
        Header: user-agent: curl/8.12.1
        Header: accept: application/vnd.elasticsearch+json; compatible-with=9
        [Full request URI: https://<<REDACTED>>.eu-west-1.aws.found.io:9243/_search]
        [Response in frame: 21]
HyperText Transfer Protocol 2
    Stream: DATA, Stream ID: 1, Length 0
        Length: 0
        Type: DATA (0)
        Flags: 0x01, End Stream
            0000 .00. = Unused: 0x00
            .... 0... = Padded: False
            .... ...1 = End Stream: True
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        [Pad Length: 0]
        Data: <MISSING>
        [Connection window size (before): 65535]
        [Connection window size (after): 65535]
        [Stream window size (before): 65535]
        [Stream window size (after): 65535]