Inotify queue overflow in file acquisition #2605
Description
We are using fsnotify/fsnotify to watch for new files if the user specifies a glob pattern.
Unfortunately, fsnotify hardcodes the events for which we are notified (see here)
Because crowdsec always adds an inotify watch on the parent directory if the user provides a glob pattern, if this directory contains a lot of files that are written constantly to, we are at risk of overflowing the kernel event queue, even though we are not interested in the events.
fsnotify has a longstanding issue about event filtering, but there does not seem to be a consensus on how it should be handled.
I see a few options available to us:
- Allow increasing the number of goroutines to handle inotify events. This feels like a hack but shouldn´t take much time to implement and likely won´t add a lot of overhead as most events will be discarded immediately (but we will still consume CPU for "nothing")
- Drop
fsnotifyand do our own minimalist implementation (the main complexity will probably be the cross-platform support) - Contribute this to
fsnotify(the better option long term , but it will probably take some back-and-forth and some time)