Skip to content

Conversation

sbalmos
Copy link
Contributor

@sbalmos sbalmos commented Aug 8, 2025

Summary

The Splunk HEC sinks run indexer ack querying on a separate spawned task, reusing the sink's original HttpRequestBuilder. However, the serialized indexer ack query bodies are raw bystreams, uncompressed. This leads to a situation where the sink, and thus the shared HttpRequestBuilder, are configured for compression, but the indexer ack query task sends uncompressed requests. This PR is a fix to clone the HttpRequestBuilder and force compression disabled, so no Content-Encoding header is sent.

A more comprehensive fix would be to pass the sink's configured Writer instance from the sink to the indexer ack task. The cloned HttpRequestBuilder in this PR would be removed, returning to using the original sink's HttpRequestBuilder. The indexer ack query request building would be reworked to compress the bytestream (if necessary) by the Writer before submitting that bytestream to the HttpRequestBuilder. Thus properly compressing the indexer ack query with the sink's configured compression.

Vector configuration

data_dir: /tmp
api:
  enabled: false
sources:
  console_in:
    type: stdin
transforms:
  parse_json:
    type: remap
    inputs:
      - console_in
    source: |-
      .message = parse_json!(string!(.message))
sinks:
  hec_out:
    type: splunk_hec_logs
    inputs:
      - parse_json
    endpoint: http://localhost:8080
    default_token: abc123
    encoding:
      codec: json
    compression: gzip
    healthcheck:
      enabled: false

How did you test this PR?

The change was tested by running against an emulated HEC via nc and manually verifying that the indexer ack query sent to the /services/collector/ack endpoint did not contain a Content-Encoding header.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • cargo fmt --all
      • cargo clippy --workspace --all-targets -- -D warnings
      • cargo nextest run --workspace (alternatively, you can run cargo test --all)
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

@sbalmos sbalmos requested a review from a team as a code owner August 8, 2025 15:50
@github-actions github-actions bot added the domain: sinks Anything related to the Vector's sinks label Aug 8, 2025
@sbalmos sbalmos changed the title fix(splunk_hec_logs splunk_hec_metrics sink): Force compression disabled in HEC indexer ack queries fix(splunk_hec sink): Force compression disabled in HEC indexer ack queries Aug 8, 2025
@sbalmos
Copy link
Contributor Author

sbalmos commented Sep 6, 2025

Poke on this. Can you also add the no-changelog label please? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: sinks Anything related to the Vector's sinks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

splunk_hec_logs sink indexer ack queries fail when using compression
1 participant