splunk_hec_logs sink indexer ack queries fail when using compression #23559
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
When using a Splunk HEC sink, namely splunk_hec_logs in my use case but this probably also affects splunk_hec_metrics, which is configured to use compression to deliver to the HEC, indexer acknowledgement queries to the HEC fail with HTTP 400s from Splunk. This makes Vector log an unuable-ack-query-response error, which then erroneously assumes event delivery success based on the original 200 from the event submission.
The root issue stems from the indexer ack query task reusing the sink's HttpRequestBuilder, which is configured to use compression. However, the indexer ack task builds the ack query request body from JSON, and sends the uncompressed bytes from this through the HttpRequestBuilder. This causes the indexer ack query HTTP request to be sent with the configured Content-Encoding header, while the request body itself remains uncompressed.
See the send_ack_query_request function in https://github.com/vectordotdev/vector/blob/master/src/sinks/splunk_hec/common/acknowledgements.rs#L212-L235, where the request is serialized by serde into bytes, a request is built by the parent HEC's HttpRequestBuilder (which would append the Content-Encoding header, even though the body bytestream is uncompressed), and then the request is sent.
Possible fix paths are to either clone the HttpRequestBuilder and explicitly set its compression property to Compression::None, or to pass along a Writer instance from the sink into the indexer ack task, so the Writer can perform the proper compression of the bytestream, before using that (compressed) bytestream in building the request.
Configuration
data_dir: /tmp
api:
enabled: false
sources:
console_in:
type: stdin
transforms:
parse_json:
type: remap
inputs:
- console_in
source: |-
.message = parse_json!(string!(.message))
sinks:
hec_out:
type: splunk_hec_logs
inputs:
- parse_json
endpoint: http://localhost:8080
default_token: abc123
encoding:
codec: json
compression: gzip
healthcheck:
enabled: false
Version
vector 0.49.0-custom-0b3c886d6 (x86_64-unknown-linux-gnu debug=full)
Debug Output
Example Data
Full capture from a real HEC was not possible as it's on a secure machine. A HEC event submission and indexer ack query was emulated using nc.
Using the above Vector config, send {"hi":true} as an event
2025-08-08T01:23:53.963293Z INFO vector::app: Log level is enabled. level="info"
2025-08-08T01:23:53.965039Z INFO vector::app: Loading configs. paths=["vector-source.yaml"]
2025-08-08T01:23:53.975012Z INFO vector::sources::file_descriptors: Capturing stdin.
2025-08-08T01:23:53.994948Z INFO vector::topology::running: Running healthchecks.
2025-08-08T01:23:53.995812Z INFO vector::topology::builder: Healthcheck disabled.
2025-08-08T01:23:53.996061Z INFO vector: Vector has started. debug="true" version="0.49.0" arch="x86_64" revision=""
2025-08-08T01:23:53.996164Z INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
{"hi":true}
2025-08-08T01:24:33.232177Z ERROR vector::internal_events::splunk_hec::sink: Unable to use indexer acknowledgements. Acknowledging based on initial 200 OK. error=ClientSendQuery error_code="indexer_ack_failed" error_type="acknowledgement_failed" stage="sending" internal_log_rate_limit=true
2025-08-08T01:24:33.523412Z ERROR vector::internal_events::splunk_hec::sink Internal log [Unable to use indexer acknowledgements. Acknowledging based on initial 200 OK.] is being suppressed to avoid flooding.
nc HEC server emulation capture:
POST /services/collector/event HTTP/1.1
content-type: application/json
authorization: Splunk abc123
x-splunk-request-channel: 48f33f52-c65c-4ea1-ba1a-cf2f54f1617c
content-encoding: gzip
accept-encoding: zstd,gzip,deflate,br
user-agent: Vector/0.49.0-custom-0b3c886d6 (x86_64-unknown-linux-gnu debug=full)
host: localhost:8080
content-length: 126
�}�[
�0н���}�� �h&S2�B ٻ��?p▒��b��M�@�RN����M2�P��̪��?�@V�*G���kg��T�^�?��Zw��4��i�����4��Y�z▒�ٕHTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 41
{"text":"Success","code":0,"ackId":12345}
POST /services/collector/ack HTTP/1.1
content-type: application/json
authorization: Splunk abc123
x-splunk-request-channel: 48f33f52-c65c-4ea1-ba1a-cf2f54f1617c
content-encoding: gzip
user-agent: Vector/0.49.0-custom-0b3c886d6 (x86_64-unknown-linux-gnu debug=full)
accept-encoding: identity
host: localhost:8080
content-length: 16
{"acks":[12345]}^C
Note above in the final chunk, the indexer ack query is posted with a content-encoding: gzip header, as the sink is configured for compression: gzip. However the ack query body itself is uncompressed.
Additional Context
No response
References
No response