Repetitious logs from vector source file

[LinTechSo]

Hi all.
I think i see some duplication with below configuration from my vector source file reading:

sources:
  my-file:
    type: file
    ignore_older_secs: 600
    include:
      - "/var/log/test.log"
    read_from: beginning

transforms:
  parse-test:
    type: remap
    inputs:
      - my-file
    source: |
       .tag = "hello"


sinks:
  log-forwarder-test:
    type: http
    inputs:
      - parse-test
    uri: https://test.com/

My problem is that the rate of my log entries in my data sources is too high from reading this log file and also I see log duplication.

I think enabling health checks or acknowledgments in my vector sink may cause this issue but haven't proven it yet. am I wrong?

Any suggestion/advice would help.

thanks.

[davidhuie-dd]

Here are a couple of things to look into:

• is your data_dir writeable? This is where Vector stores its file checkpoints for reading through the file: https://vector.dev/docs/reference/configuration/sources/file/#data_dir
• you're correct that the healthcheck & acks could be causing issues like this. Ensure that the HTTP server is responding with a 2xx response code to both the health check & event sink endpoints

If you're still experiencing issues, please provide the Vector logs 😄