Vector not accepting connections on sink failure

[jakegut]

I'm using Vector v0.35.0 in an aggregator configuration, where it accepts json via HTTP, from FluentBit agents, and inserting it into ClickHouse using the sink. I'm hosting it on AWS ECS Fargate in front of an NLB with health-checks enabled based on the API /health endpoint.

Whenever the sink starts to experience a large amount of failures, in my case a number of "connection resets" from ClickHouse Cloud whenever it scales up, Vector will soon stop accepting connections resulting in an unhealthy task and being killed off by AWS. Vector also shoots up in memory, but there's nothing indicating in the logs (level=info) to indicate that.

I tried buffering to filesystem to see how it would perform and it resulted in a 2-3x performance decrease, which is a considerable amount at our scale, so I haven't tried that out with ClickHouse Cloud scaling up.

Example error:

2024-02-06T08:57:49.515492Z  WARN sink{component_kind="sink" component_id=clickhouse_envoy component_type=clickhouse}:request{request_id=9069}:http: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing" internal_log_rate_limit=true

Vector config:

api:
  enabled: true
  address: "0.0.0.0:8686" 

sources:
  envoy_in:
    type: http_server
    address: 0.0.0.0:8123
    encoding: ndjson
    path_key: __vector_http_path

sinks:
  clickhouse_envoy:
    type: clickhouse
    inputs:
      - envoy_in
    endpoint: https://<host>.us-west-2.vpce.aws.clickhouse.cloud:8443
    database: default
    table: envoy
    auth:
      strategy: basic
      user: default
      password: <password>
    batch:
      max_bytes: 30000000
      timeout_secs: 2
# -- later tried --
    buffer:
      type: disk
      max_size: 4294967296 # 4 GB    

[jszwedko]

The behavior you are experiencing does sounds expected to me. When the clickhouse sink cannot successfully send data it'll start applying back-pressure to the source (in this case the http_server source). The http_server source, in the presence of backpressure, will accept requests, but hold them open until it can flush the data. It sounds like you hit a cap on the number of open requests where Vector stops responding.

A few other ideas you could try:

• Use a large in-memory buffer to tolerate the Clickhouse downtime (shouldn't have the performance downside of a disk buffer)
• Configure the buffer to drop (buffer.on_full: "drop_newest") to shed load when data cannot be sent
• Decrease the retry interval in the clickhouse sink if you think it'll quickly recover (by default Vector will exponentially back-off retry requests)

[jakegut]

... will accept requests, but hold them open until it can flush the data. It sounds like you hit a cap on the number of open requests where Vector stops responding

This explains a lot. Based on Fluent-bit's docs and some local testing, FB will create a new request if there's a new chunk ready to flush and none of the other connections have received a response, and it will create an unlimited number of requests.

I've been experimenting with FB's and Vector Forward protocol and FB will not create an unlimited number of requests with that, (at least when Require_ack_response=True). It makes sense to use the Forward I/O in this case.

I'll try suggestion 1 and 3, 2 is not an option for us at the moment but I'll keep that in mind. I was checking the Buffer docs and I saw there was an overflow option, I'll keep an eye on it for when it's production ready.