Vector Back-Pressure Causing Severe Delays (10–15 Days) in Log Ingestion from ~1200 Windows Hosts

[greycel]

Question

Hi Everyone,

We are receiving data that is 10–15 days old and observing persistent errors in our Vector deployment:
Error writing acknowledgement, dropping connection (occurs every few seconds)
i/o error: Operation timed out (os error 110) (intermittent)
These errors typically appear 5–10 minutes post-restart.

Environment:
Hardware: 16vCPU, 32GBRAM
Vector version: vector 0.46.0 (x86_64-unknown-linux-musl a7429e2 2025-04-08 16:05:37.911631718)
Winlogbeat version: 6.7.1

~1200 Windows hosts forwarding logs via Winlogbeat (version 6.7.1) → Vector (0.46.0)
Vector configured to parse Windows logs and forward to an AWS S3 sink
Sink configured with a 20GB disk buffer

Observed behavior:
After restarting the Vector service, we observe the following connection behavior (via ss -s):
Immediately post-restart: ~700 ESTAB connections, higher than CLOSED connections
Within 5–10 minutes: ESTAB connections drop to ~360 and fluctuate in the 350–380 range, and CLOSED connections at 3300.

Appears like Vector is unable to sustain the initial connection load from ~1200 Windows hosts. Eventually, back-pressure starts to build at the S3 sink and propagate upstream through Vector to Winlogbeat. This results in log ingestion delays, with Vector tap receiving data that is 10–15 days old.

We have tuned the OS-level buffers and limits as below:

## /etc/security/limits.conf
## System Limits for FDs
## "nofile" is "Number of Open Files"
## This is the cap on number of FDs in use concurrently.
## Set nofile to the max value of 1,048,576.
#<user>     <type>    <item>     <value>
*           soft      nofile     1048576
*           hard      nofile     1048576
root        soft      nofile     1048576
root        hard      nofile     1048576

### /etc/sysctl.conf
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 33554432 134217728
net.ipv4.tcp_wmem = 4096 33554432 134217728
net.core.netdev_max_backlog = 15000
net.core.somaxconn = 8192
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_probes = 6
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_congestion_control = bbr
net.ipv4.ip_local_port_range = 10000 65535
net.ipv4.tcp_retries2 = 5
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_abort_on_overflow = 1
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_syncookies = 1
fs.file-max = 1048576

Vector Config

sinks:
  output_s3:
    acl: "private"
    auth:
      access_key_id: "access_key"
      region: "us-east-1"
      secret_access_key: "secret_access_key"
    batch:
      max_bytes: 15000000
      max_events: 5000
      timeout_secs: 30
    bucket: "data-logs"
    buffer:
      type: "disk"
      when_full: "block"
      max_size: 21474836480
    compression: "gzip"
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"
    inputs:
    - "enrichments"
    key_prefix: "logs/{{ .organization.id }}/{{ .log.type }}/%Y/%m/%d/"
    region: "us-east-1"
    storage_class: "STANDARD"
    type: "aws_s3"
sources:
  input_windows:
    address: "0.0.0.0:12102"
    connection_limit: 4000
    keepalive:
      time_secs: 60
    type: "logstash"
transforms:
  preprocess_microsoft_windows:
    inputs:
    - "input_windows"
    source: |-
      .log.schema = "raw"
      .log.format = "json"
      .log.source_type = "winevtlogs"
    type: "remap"

Vector Logs

Sep 10 06:33:20 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:20.921062Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] is being suppressed to avoid flooding.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.3.14.4:52596","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.3.14.4:52596","name":"connection"}]}
Sep 10 06:33:30 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:30.606066Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] has been suppressed 122 times.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.173.16.20:60432","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.173.16.20:60432","name":"connection"}]}
Sep 10 06:33:30 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:30.606114Z","level":"ERROR","message":"Error writing acknowledgement, dropping connection.","error":"Broken pipe (os error 32)","error_code":"ack_failed","error_type":"writer_failed","stage":"sending","internal_log_rate_limit":true,"target":"vector::internal_events::tcp","span":{"peer_addr":"10.173.16.20:60432","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.173.16.20:60432","name":"connection"}]}
Sep 10 06:33:30 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:30.630351Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] is being suppressed to avoid flooding.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.124.110.63:55383","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.124.110.63:55383","name":"connection"}]}
Sep 10 06:33:40 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:40.629455Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] has been suppressed 109 times.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.3.7.76:65344","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.3.7.76:65344","name":"connection"}]}
Sep 10 06:33:40 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:40.629495Z","level":"ERROR","message":"Error writing acknowledgement, dropping connection.","error":"Broken pipe (os error 32)","error_code":"ack_failed","error_type":"writer_failed","stage":"sending","internal_log_rate_limit":true,"target":"vector::internal_events::tcp","span":{"peer_addr":"10.3.7.76:65344","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.3.7.76:65344","name":"connection"}]}
Sep 10 06:33:40 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:40.828824Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] is being suppressed to avoid flooding.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.173.16.68:50944","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.173.16.68:50944","name":"connection"}]}
Sep 10 06:33:50 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:50.741727Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] has been suppressed 107 times.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.170.8.101:54278","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.170.8.101:54278","name":"connection"}]}
Sep 10 06:33:50 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:50.741763Z","level":"ERROR","message":"Error writing acknowledgement, dropping connection.","error":"Broken pipe (os error 32)","error_code":"ack_failed","error_type":"writer_failed","stage":"sending","internal_log_rate_limit":true,"target":"vector::internal_events::tcp","span":{"peer_addr":"10.170.8.101:54278","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.170.8.101:54278","name":"connection"}]}
Sep 10 06:33:50 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:50.789318Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] is being suppressed to avoid flooding.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.1.207.41:49483","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.1.207.41:49483","name":"connection"}]}
Sep 10 06:33:58 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:33:58.065981Z","level":"ERROR","message":"Failed framing bytes.","error":"i/o error: Operation timed out (os error 110)","error_code":"decoder_frame","error_type":"parser_failed","stage":"processing","internal_log_rate_limit":true,"target":"vector::internal_events::codecs","span":{"peer_addr":"10.126.110.19:64053","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.126.110.19:64053","name":"connection"}]}
Sep 10 06:34:00 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:34:00.758432Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] has been suppressed 112 times.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.2.201.18:55281","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.2.201.18:55281","name":"connection"}]}
Sep 10 06:34:00 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:34:00.758476Z","level":"ERROR","message":"Error writing acknowledgement, dropping connection.","error":"Broken pipe (os error 32)","error_code":"ack_failed","error_type":"writer_failed","stage":"sending","internal_log_rate_limit":true,"target":"vector::internal_events::tcp","span":{"peer_addr":"10.2.201.18:55281","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.2.201.18:55281","name":"connection"}]}
Sep 10 06:34:01 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:34:01.371339Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] is being suppressed to avoid flooding.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.124.110.201:62450","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.124.110.201:62450","name":"connection"}]}
Sep 10 06:34:10 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:34:10.870518Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] has been suppressed 97 times.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.168.72.23:55613","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.168.72.23:55613","name":"connection"}]}
Sep 10 06:34:10 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:34:10.870563Z","level":"ERROR","message":"Error writing acknowledgement, dropping connection.","error":"Broken pipe (os error 32)","error_code":"ack_failed","error_type":"writer_failed","stage":"sending","internal_log_rate_limit":true,"target":"vector::internal_events::tcp","span":{"peer_addr":"10.168.72.23:55613","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.168.72.23:55613","name":"connection"}]}
Sep 10 06:34:11 BLULC01 vector[373078]: {"timestamp":"2025-09-10T05:34:11.081175Z","level":"ERROR","message":"Internal log [Error writing acknowledgement, dropping connection.] is being suppressed to avoid flooding.","target":"vector::internal_events::tcp","span":{"peer_addr":"10.3.10.121:52428","name":"connection"},"spans":[{"component_id":"input_windows","component_kind":"source","component_type":"logstash","name":"source"},{"peer_addr":"10.3.10.121:52428","name":"connection"}]}

[pront]

Hi @greycel, before diving deep into the implementation details, two things stood out:

• your winlogbeat version is quite old, did you consider upgrading to a more recent version with improved connection stability?
• you can introduce 2+ Vector instances here and a load balancer in front of them
  • or run local aggregator before shipping to Vector, this is applicable only if Vector can handle the load (to determine this we would have to examine Vector's internal metrics)

[greycel]

Hi @pront ,

• Yes, all our systems are currently running an older version of Winlogbeat. We're planning to update to a more recent version, which should help improve connection stability.

• We're currently setting up a test environment with two containerized Vector instances behind a load balancer to evaluate performance. Given that we have only one physical instance available, our plan is to run them as Docker containers.

Would you prefer deploying Vector as Docker containers, or would you prefer provisioning them as separate systems instead?
Also, would containerized Vector instances be sufficient to handle the expected load?

[pront]

Would you prefer deploying Vector as Docker containers, or would you prefer provisioning them as separate systems instead?

We generally do not recommend anything about deployments. Each infrastructure/environment is unique and we don't have access to it. I would start with the easiest option and explore alternatives if problems are discovered.

Also, would containerized Vector instances be sufficient to handle the expected load?

My guess is that yes, if you distribute the 1,200 clients across multiple Vector instances and ensure fast local storage for the disk buffers. But, there's only way to be sure.