Vector does not appear to work with Python's RotatingFileHandler

[shomilj]

Has anybody used RotatingFileHandler with Vector? We're hitting an issue where it appears that Vector is holding references to open file handles (similar issue to #11742) - here is some output from lsof ~

vector        951                              root 3602r      REG              259,1   38322044     535038 /tmp/ray/session_2023-04-27_13-06-49_673121_114/logs/monitor.log (deleted)
vector        951                              root 3603r      REG              259,1   40118317     535671 /tmp/ray/session_2023-04-27_13-06-49_673121_114/logs/monitor.log (deleted)
vector        951                              root 3608r      REG              259,1   44385144     537108 /tmp/ray/session_2023-04-27_13-06-49_673121_114/logs/monitor.log (deleted)
vector        951                              root 3609r      REG              259,1   49535510     538835 /tmp/ray/session_2023-04-27_13-06-49_673121_114/logs/monitor.log (deleted)

[spencergilbert]

That does seem similar, could you try to strace to see if Vector is still reading from those (as suggested in that issue)?

[shomilj]

I tried it and got nothing (no output). Anything else I should run?

[spencergilbert]

🤔 not off the top of my head, it does seem like the same issue that you linked - I suspect we need to find some time to investigate the source for a while to uncover the root cause.

[shomilj]

Ah, ok. Any workarounds? Are there other log rotation strategies that might be able to avoid this? We could try switching to the standard linux logrotate instead of the Python application layer RotatingFileHandler (that's next on my list) -- but if it has the same issue I don't know how we can work around this other than killing Vector every few hours 😭

[spencergilbert]

I think the only thing that would be super helpful is if you could reliably reproduce this behavior, that would definitely help us get a fix/find a workaround.