Don't understand blocking disk buffer memory usage behavior

[a-abella]

Hello, I've been investigating Vector for a use-case that I'm surprised a dedicated solution didn't exist (or at least, none that I found): a persistent disk-buffered HTTP POST proxy with all the retry and backoff bells and whistles.

In my configuration I have http_server sources that accept arbitrary json and protobuf payloads without decoding. I set the event body to the raw bytes in a remap, then I send the raw bytes to an http sink with appropriate content-type headers with a blocking disk buffer. I also have acknowledgements enabled.

This is pretty-much working, but I was confused by a memory-usage behavior that occurs when the buffer fills and begins to block. Vector's memory usage climbs as though it's spilling over to a secondary in-memory buffer, but I don't have multiple buffers configured.

While the memory usage grows, the http_server source is responding with 5XX to my clients. So despite responding with an error, it actually seems like it's accepting the payload and storing in memory.

The http sink does receive and attempt to post the events that are in memory. Once the backend services stop back-pressuring, memory usage does drop as events are sent and accepted. So it is working exactly like a buffer.

I enabled the --allocation-tracing command line argument to understand which component was consuming memory, as I understand there are memory buffers between components in the pipeline. What I found is that the memory usage is allocated to component_id="root", not any user-defined component:

By comparing timestamps it's visible that that memory usage began increasing when the buffer growth plateaued due to reaching its 2GB size:

This is the complete config for one HTTP source-sink pipeline I've been testing (configured via helm):

  acknowledgements:
    enabled: true

  sources:
    src_tempo_nonprod:
      type: http_server
      address: 0.0.0.0:4418
      decoding:
        codec: bytes
      framing:
        method: bytes
      keepalive:
        max_connection_age_jitter_factor: 0.1
        max_connection_age_secs: 300

  transforms:
    remap_tempo_nonprod: &remap_body
      type: remap
      inputs: [src_tempo_nonprod]
      source: |
        . = .message

  sinks:
    sink_tempo_nonprod: &dst_tempo_raw
      inputs: [remap_tempo_nonprod]
      type: http
      request:
        headers:
          X-Scope-OrgID: "{{ .Values.tenant_id_nonprod }}"
          Content-Type: application/x-protobuf
        rate_limit_num: 5
      uri: "http://tempo-distributor.tempo.svc.cluster.local:4318/v1/traces"
      compression: none
      batch:
        # max_events must be 1 because data is already batched by the time it reaches Vector,
        #  also, since we're handling raw bytes, we can't split or concatenate events without
        #  corrupting the byte stream
        max_events: 1
      encoding:
        codec: raw_message
      framing:
        method: bytes
      buffer:
        type: disk
        max_size: 2_147_483_648 # 2GB
        when_full: block
      acknowledgements:
        enabled: true

I suppose my expectation would have been that with a full buffer, blocking, and acknowledgments on, the http_server sources should start rejecting client payloads without any kind of memory buffering.

I'm not sure if this is working as designed, or if I've tripped and stumbled on something unexpected.

[pront]

Hi @a-abella, apologies for the delayed response—this one slipped through the cracks. You’ve provided sufficient information; we just need to carve out some time to investigate what’s happening. In the meantime, please don’t hesitate to reach out or share any additional details.

[a-abella]

Hi @pront, glad to hear this got picked up.

Since this post I've been running with disk buffers set to drop_newest, but I can reproduce if needed.

[pront]

Since this post I've been running with disk buffers set to drop_newest

Did memory usage patterns change with this new setting?

Linking here some similar issues related to memory consumption:

• Memory usage spikes + OOM kills despite minimal config #22893 (comment) (this might be relevant here)
• Slow memory leak in vector aggregate #23093

[a-abella]

@pront Yes, in drop_newest mode the memory growth observed when the dusk buffer is full does not occur. I've only seen it under the following conditions:

• disk buffer is configured to block
• disk buffer is full, thus blocking