Nxlog doesn't retry missing files

[dpb587]

Sometimes on new machines nxlog may start before log writers have had a chance to touch their initial files. If nxlog is configured to use a plain path, it emits a warning about /var/path/stdout.log does not exist and doesn't seem to retry. I tried switching to a prefix glob (e.g. /var/path/*stdout.log) to take advantage of how globs are supposed to check for new files, but it emits Module abcd has no input files to read and doesn't appear to check for new files again. The glob prefix method is what's currently committed.

Not sure what other things to try. I don't want the config generator to be touching all files to ensure they exist because it's not aware of what permissions/owners they and any intermediate directories should be.

Current solutions are to manually restart the logsearch-logs service, or use a sufficiently large logsearch.logs.start_delay property.

[dpb587]

@mrdavidlaing, any extra insight from your experience with nxlog?

[mrdavidlaing]

@dpb587, that is weird; cause I've definately seen nxlog on windows repeatedly moan about missing files everytime it polls them for changes.

Have you tried enabling NXLOGs DEBUG log level, and logging it to its own log file (rather than STDOUT)? I've noticed that some errors don't get written to STDOUT when running in -f mode.

[mrdavidlaing]

Here is a pretty useful NXLOG debugging config - just don't leave it running too long since nxlog.log gets HUGE very quickly:

LogLevel    DEBUG
LogFile     nxlog.log

...snip...

<Output out_file>
    Module  om_file
    File    "out.log"
</Output>

<Input in_file>
    Module  im_file
    File    "/var/path/*stdout.log"
</Input>

<Route route>
    Path in_file => out_file
</Route>

[mrdavidlaing]

Finally, make sure you are running the latest nxlog-ce-2.8.1248, which I note contained the following in its release notes

A regression caused a crash with im_file when the File did not exist.

[dpb587]

I increased the default log forwarding startup delay from 10s to 60s. At a minimum, this will most certainly detect new files at the cost of slightly delayed initial log forwarding. I also enabled LogFile, so we're at least logging as much as we practically can.

I did re-test the file behaviors and in theory it seems like nxlog should be noticing new files. I did notice a backoff behavior when the file didn't exist at startup, but it just meant the file took longer to get forwarded and it was eventually added. So, I've either confused myself with the number of dev versions we have deployed, or there's something else amiss that the additional logging should help us find.

In the meantime, I've deployed these latest changes to our deployments, restarted many nodes, and this is now at least behaving better than it was.

[dpb587]

We should consider managing a cron job which regularly regenerates the config at a configurable interval and SIGHUP the shipper to pick up new files. We avoid writing globs to the shipper config because it becomes difficult to respect disabled files and altered fields configured in the various override properties.