Filebeat 9.x bulk requests contain double newline, breaking ES-compatible endpoints

[trilamsr]

Description

Filebeat 9.x produces NDJSON bulk request bodies with double newlines (\n\n) between events. While Elasticsearch tolerates empty lines, ES-compatible endpoints (Axiom, OpenSearch) reject them:

400 Bad Request: invalid event at index 1: ReadObject: expect { or , or } or n, but found \u0000

Root cause

In 9.x, events are pre-encoded in the pipeline queue (event_encoder.go). The Marshal/AddRaw call appends a trailing \n to the encoded bytes. When the bulk body assembler later writes these bytes via AddRaw(RawEncoding{...}), it unconditionally appends another \n, producing an empty line.

This didn't exist in 8.x because events were encoded inline during bulk assembly (no RawEncoding path).

Steps to reproduce

1. Configure filebeat-oss:9.3.1 with a filestream input and output.elasticsearch pointing to any non-Elasticsearch endpoint (e.g. Axiom's ES compatibility API)
2. Ingest a log file
3. Observe 400 Bad Request with null byte error on every bulk write
4. Same config with filebeat-oss:8.11.1 (using container input) works fine

Affected versions

• 9.0.0 through 9.3.1 (current latest)
• Both -oss and standard builds
• Both jsonEncoder and gzipEncoder paths

Fix

PR #49557 — checks if RawEncoding bytes already end with \n and skips the extra one.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)