Reading log files from the beginning just once

[gingerbreadtrev]

I want to use vector to replace filebeat but I haven't been able to find a way to make vector replicate filebeat's ability to read through log files starting at the beginning and then exit when reaching the EOF.

I have a static set of logs won't have anymore logs added so I don't need to tail or continuously monitor them I just need to read through them all once. I wanted to try using vector because of it's transformation abilities so I was wondering if this was possible?

[pront]

Hi @gingerbreadtrev, I believe you are looking for the file source, and setting read_from to beginning.

Also, you might be interested in the remove_after_secs option.

After reaching EOF, the number of seconds to wait before removing the file, unless new data is written.

If not specified, files are not removed.

[gingerbreadtrev]

thanks for reaching out! i've been playing with removing the files after reading them and i think that'll solve my problem. right now i've got a separate script watching the directory for all the files to be removed and then ending the vector process when they are.

i just want to make sure im not missing anything, is there any better or built in way to make vector terminate when it's done reading everything?

[pront]

is there any better or built in way to make vector terminate when it's done reading everything?

You will have to terminate the Vector instance manually. Typically we want the instance to stay up and running to process new files as they come in.