Only read specific lines when using file source

[arch-xtof]

Hello! I have files containing logs similar to this:

| 105537091 | x.x.x.x | 3306 | 0    | 1000   | 0      | 0   | 100       | 5       | 0   | 0       |                                 | 140322404325440 |
| 105536652 | x.x.x.x | 3306 | 0    | 1000   | 0      | 0   | 100       | 5       | 0   | 0       |                                 | 140322404322560 |
+-----------+---------+------+------+--------+--------+-----+-----------+---------+-----+---------+---------------------------------+-----------------+
2023-09-21 11:03:10 MySQL_Thread.cpp:3910:process_all_sessions(): [WARNING] Closing unhealthy client connection x.x.x.x:yyyy
2023-09-21 11:03:14 MySQL_Thread.cpp:3910:process_all_sessions(): [WARNING] Closing unhealthy client connection x.x.x.x:yyyy

I am using a file source and wondering if a vector can only read lines containing the [WARNING] string. I know I can use a filter or transform remap to achieve this, but I am interested if I can ignore other lines at the very source.

Thanks for looking into this in advance!

[dsmith3197]

Hi @arch-xtof,

I don't believe this is possible today. I'm curious are there reasons why including a filter transform after the source is not sufficient?

[arch-xtof]

I just thought it would go easier on resources, that's the only reason.