Does the File source use poll or notify?

[WuBingzheng]

I want to use File source to collect logs with low latency. However there are random several seconds (about 1s ~ 10s) delay on new contents in the log file.

Here is my configuration file. Is there anything that I miss?

data_dir = "."

[sources.file]
type = "file"
include = [ "tmp/test.log" ]
encoding.charset = "utf8"

[sinks.print]
type = "console"
inputs = ["file"]
encoding.codec = "text"

I use Ubuntu 16.04.

[jszwedko]

Hi @WuBingzheng ,

The file source does polling to pick up new files. By default it scans every second but this is configurable via https://vector.dev/docs/reference/configuration/sources/file/#glob_minimum_cooldown_ms

[WuBingzheng]

This glob_minimum_cooldown_ms specifies the interval to pick up new files. But what I want is to pick up new logs in the existing files.

[jszwedko]

Ah, I misunderstood. Vector will attempt to scan each file "fairly" in that it'll read a bit from each file in rotation. https://vector.dev/docs/reference/configuration/sources/file/#max_read_bytes can be used to control how many bytes are read at a time from a given file.

[WuBingzheng]

Thanks for your reply, but I think that's not what I want still. I do not care about 'fair', there is only 1 source file. I want low latency. I want vector to pick up the new log immediately.

My original scene is: one program generates logs into a file; and vector picks up the logs and pushes them into kafka; some other applications subscribes from kafka. I want this message stream to be low latency. However the vector picks up logs with a high random latency (about 1s - 10s).

I used telegraf now, which uses inotify as default and provides a very low latency. Now I want to replace it by vector, but the high latency is not acceptable.

program -> telegraf -> kafka -> apps
|<------- 3ms latency ------------>|

program -> vector -> kafka -> apps
|<-- 1s~10s -->|

[jszwedko]

Hmm, for one source file there shouldn't be a difference; it should continuously be trying to read the input file. I would suggest trying an increase to https://vector.dev/docs/reference/configuration/sources/file/#max_read_bytes and see if that helps.

[WuBingzheng]

max_read_bytes does not help.
Thanks anyway.

[WuBingzheng]

I read the source code and find that vector polls the file actually. And to avoid burning the CPU, it sleeps some time after each polling. Here is the code.

Why does it polls? Why does it not just read the file after being notified (e.g. by inotify in Linux)?

[jszwedko]

Hi @WuBingzheng !

The initial implementation of the file source used a polling mechanism rather than inotify for easier cross-platform support and avoiding inotify flakiness (undelivered notifications). We have discussed using inotify to augment the discovery process on Linux though (used in addition to polling).

[WuBingzheng]

Thanks.

The file source now uses a hard-code sleep time cap (2 seconds).
But I want a much smaller cap, so I change this cap to configurable. Is this change welcomed as a PullRequest?

[jszwedko]

Hi @WuBingzheng ! I think we'd be open to a PR making that configurable.