Persistent storage to protect against data loss

[mihaiste]

Hello everyone,

We have an OpenTelemetry collector deployed on a Kubernetes cluster that receives metrics on 4317 (from another OpenTelemetry collector instance) and sends them to our storage backend (Victoria Metrics).

We have the following configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "${env:MY_POD_IP}:4317"
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 70
    spike_limit_percentage: 10
  batch:
    send_batch_size: 2048
    timeout: 200ms
    send_batch_max_size: 4096
  deltatocumulative:
    max_stale: 5m
exporters:
  otlphttp:
    compression: gzip
    encoding: proto
    auth:
      authenticator: basicauth
    endpoint: "https://victoria-metrics-auth"
    sending_queue:
      storage: file_storage
    retry_on_failure:
      enabled: true
      initial_interval: 15s
      max_interval: 60s
      max_elapsed_time: 10m
extensions:
  file_storage
    directory: /var/lib/otel/buffer
    create_directory: true
    timeout: 10s
    compaction:
      directory: /var/lib/otel/buffer
      on_rebound: true
      rebound_needed_threshold_mib: 100
      rebound_trigger_threshold_mib: 10
      check_interval: 5s
  basicauth
    client_auth:
      username: ${env:BASIC_AUTH_USERNAME}
      password: ${env:BASIC_AUTH_PASSWORD}
service:
  extensions:
  - basicauth
  - file_storage
  pipelines:
    metrics:
      receivers:
      - otlp
      processors:
      - memory_limiter
      - deltatocumulative
      - batch
      exporters:
      - otlphttp

The question is in regards to buffering the data outgoing to Victoria Metrics: we want to set up buffering to persistent storage only.
From the metrics we observed in Victoria Metrics we understand the fact that there is a limitation of queue capacity to 1000, although it is not present in the configuration. Also, the persistent disk (configured to 20GB) is not fully in use.
Is this by design? Is the queue_size parameter required in the configuration in order to use the entire disk for buffering?