How to Define Disk Buffer Storage Location and Customize the Path in Log Sinks?

[hamedsa-78]

When we define the buffer type as 'disk,' where are the buffered logs stored? What is the default path, and how can we configure a custom path?

sinks:
  overflow_test:
    type: blackhole
    buffer:
    - type: memory
      max_events: 1000
      when_full: overflow
    - type: disk
      max_size: 1073741824 # 1GiB.
      when_full: drop_newest

[hamedsa-78]

I configured this, and the buffer folder was created in the /vector-data-dir directory. However, even after successful delivery, the buffer retains its data and is not flushed. With this configuration, it seems the buffer will eventually fill up. Am I correct? Any solution for this?

customConfig:
  data_dir: "/vector-data-dir/"
  api:
    enabled: true
    address: 127.0.0.1:8686
    playground: false
  sources:
    file_logs:
      type: "file"
      include:
      - "/vector-data-dir/log.log"
      read_from: "beginning"
  transforms:
    app_logs_parser:
      inputs:
      - "file_logs"
      type: "remap"
      source: |
        # Try to parse the JSON message
        . = parse_json(.message) ?? null

        # Check if parsing was successful
        if is_null(.) {
          abort # unknown type
        }

        # Handle two formats of logs: flat and nested
        log_value =
          if exists(.log) {
            .log
          } else {
            .
          }

        # Merge log_value into the main object and add timestamp
        . = log_value
  sinks:
    kafka_server:
      type: "kafka"
      inputs:
      - "app_logs_parser"
      encoding:
        codec: "json"
      bootstrap_servers: kafka.vector.svc.cluster.local:9092
      topic: test
      buffer:
        type: disk
        max_size: 268435488 # 256MB
        when_full: block

[jszwedko]

Full data files should be deleted after completely flushed.

[hamedsa-78]

Full data files should be deleted after completely flushed.

is this the answer of our question from documentation?
Additionally, like in-memory buffers, disk buffers have a configurable maximum size so they can be limited in terms of disk usage. This maximum size is adhered to rigidly, so you can depend on Vector not exceeding it. There is a minimum size for all buffers, though – currently ~256MiB – which is a requirement of the disk buffer implementation. On the filesystem, disk buffers will look like append-only log files that grow to a maximum file size of 128MiB and are deleted once they have been processed fully.

[hamedsa-78]

Full data files should be deleted after completely flushed.

In my case actually the data has been sent to Kafka, but the buffer size does not change and it has not flushed.

[jszwedko]

Hey! Yes, what you found in the documentation is correct. Once all records in a given data file have been sent by the sink, the data file will be deleted and a new one created (iirc).

[hamedsa-78]

Yes, I tested it. The buffer for slink grows up to 128 MB before it's flushed. Even if slink successfully sends data to the destination (Kafka in our case), the buffer can still reach 128 MB before clearing.