how fluentd ensure collect docker logs completely when container quit

[wangchaoforever]

When a container quit, it's log file will lose also. So my question is when the container's log lose, how fluentd ensure collect the log completely?

[fujimotos]

So my question is when the container's log lose

Generally speaking, system delays the actual file deletion until the last
open file handler is closed. The exact semantics varies depending on OS,
but it's (mostly) true on both Unix and Windows.

Here is what happens to Fluentd on file deletion:

• Fluentd keeps open file handlers to log files that it monitors.

• When Fluentd notices that some file was deleted, it thinks
  "hmm, the log file must be rotated by some other program".

• Fluentd switches to a rotate mode, and keeps reading from the deleted
  file for 5 seconds.

• Then close the file handler for the deleted file.

This generally works, and Fluentd can read up to the last line of
deleted files.

[wangchaoforever]

@fujimotos thanks for your answer. I learned a lot from your answer. But I have another problem: when there are too many log files for fluentd to monitor and even some container log files had no time to enter the monitoring list when they quit. I think those container's log will lose and how to resolve this ？

[fujimotos]

when there are too many log files for fluentd to monitor and even some container log files had no time to enter the monitoring list when they quit

Hmm. I think there is no perfect solution for that issue.
Here are some directions you can pursue:

• Use multiple workers to monitor files.
  • Instead of reading everyting in a single process, partition the
    files into multiple processes.
  • This should prevent Fluentd from being overloaded by too many files,
    and thus reduce the probability of missing ephemeral files.
• Use Docker's logging driver
  • Insted of reading from files, let Docker forward log data to
    Fluentd.

See the following docs for details on both options:

• https://docs.fluentd.org/deployment/multi-process-workers?q=worker
• https://docs.docker.com/config/containers/logging/fluentd/

[wangchaoforever]

@fujimotos thanks a lot, I see.