filter_grep: grep with arithmetic operators

[marcosdiez]

Documentation: fluent/fluent-bit-docs#810

The grep filter does uses regexes only to match strings.
So I added a few arithmetic operators to match numbers (booleans, ints, floats and doubles):

• number_equal
• number_not_equal
• number_less_than
• number_less_than_or_equal
• number_greater_than
• number_greater_than_or_equal

Simple example to test:

[SERVICE]
    flush        1
    daemon       Off
    parsers_file parsers.conf
    plugins_file plugins.conf

[INPUT]
    Name         dummy
    dummy        {"message":"dummy", "aaaa": "bbb", "cccc": 128, "ddd": 20}


[FILTER]
    name    grep
    match   *
    # number_greater_than   ddd 40 # will not display any output
    number_greater_than   ddd 10 # will     display   output

[OUTPUT]
    name  stdout
    match *

debug output: https://gist.github.com/marcosdiez/52f1aca31ac69dda43c3dbe2d56e4ad0
Valgrind output: https://gist.github.com/marcosdiez/7f37b3e53a5252c2c33569e01e5b79d2

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[nokute78]

I think it is better to create new filter plugin to support these operators.
Because grep command doesn't support these operations.

[marcosdiez]

Hi I thought about that. But then we would have two different commands that with the same final objective: drop arbitrary log entries. Also, I am not sure what we should call it. Numerical Parser? Numerical Evaluator? Algebric log entry remover? Number Comparer? I considered also not adding the arithimethical methods and just silently converting the numbers to a string so they would work with a regular expression, but I thought that would be more restrictive and way less efficient. One last argument against creating a new filter is that we already have "Modify" and "Record Modifier" which are confusing enough and parially redundant. I did not want to go that route. I do agree the interface is not perfect and we could improve. On the bright side, the code is already processing GBs of data daily and working well :) Il 14 Maggio 2022 09:24:53 GMT-03:00, Takahiro Yamashita ***@***.***> ha scritto:

I think it is better to create new filter plugin to support these operators. Because `grep` command doesn't support these operations. -- Reply to this email directly or view it on GitHub: #5449 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[agup006]

I'm +1 to having this in grep filter for now. We most likely need a longer-term implementation on comparisons (str + numbers). Though I don't think it should block progress for solving the current challenge. cc @nokute78

[nokute78]

The basis of grep is match or not match pattern.
https://linux.die.net/man/1/grep

grep searches the named input FILEs (or standard input if no files are named, or if a single hyphen-minus (-) is given as file name) for lines containing a match to the given PATTERN.

I agree to support to check if the number is matched or not.

On the other hand, comparison operation is not supported by grep.
Such operations are supported by awk and so on.
That's why I requested to create new plugin.

One last argument against creating a new filter is that we already have "Modify" and "Record Modifier" which are confusing enough and parially redundant.

I agree and it is complicated for user...
But I think Comparing number and Matching pattern are different functionalities.

I am not sure what we should call it. Numerical Parser? Numerical Evaluator? Algebric log entry remover? Number Comparer?

Naming is one of difficult things.
I'm not familiar with English, so I can't help it :( ...
filter_compare_num like filter_rewrite_tag ? Hmm..

[patrick-stephens]

See fluent/fluent-bit-docs#810

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[eschabell]

@marcosdiez can you take a look at resolving the conflicts in this PR?