processor_calyptia: new lua processor for logs/metrics/traces

[tarruda]

The new processor has a similar API to filter lua, with the following differences:

• logs/metrics/traces are supported
• it is now possible to modify logs metadata
• when splitting logs, it is possible to also specify the timestamp/metadata of each split log.
• global variables for the return codes were added, these can make scripts more readable. The variables are:
  • DROP (-1)
  • KEEP (0)
  • MODIFY (1)
  • MODIFY_KEEP_TIMESTAMP (2)
• For metrics/traces, timestamp is not used (since timestamps are specified using internal fields)
• For metrics, metadata is not used for now

NOTE: I could not use lua as the processor name because it would conflict with filter_lua, and filters have a precedence when instantiating a processor. Since this processor is used in the implementation of calyptia processing rules, I thought it would make sense to name it "calyptia".

This processor is already fully tested as part of core fluent-bit project. The tests make use of the integration tests infrastructure of calyptia core. I have added the tests in the examples directory, since these tests can also be used as API examples. After building, you can run the examples like this:

/build/bin/fluent-bit -c examples/processor_calyptia/logs_modify_key/fluent-bit.yaml

*update: I have added a test script under the "tests" directory. This script will run every examble in the examples/processor_calyptia directory with valgrind (each one runs as a background job). These are the valgrind results of running locally:

==63736== Memcheck, a memory error detector
==63736== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63736== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63736== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/logs_empty_key/fluent-bit.yaml
==63736== Parent PID: 63735
==63736== 
==63736== 
==63736== HEAP SUMMARY:
==63736==     in use at exit: 0 bytes in 0 blocks
==63736==   total heap usage: 4,472 allocs, 4,472 frees, 5,403,388 bytes allocated
==63736== 
==63736== All heap blocks were freed -- no leaks are possible
==63736== 
==63736== For lists of detected and suppressed errors, rerun with: -s
==63736== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==63737== Memcheck, a memory error detector
==63737== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63737== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63737== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/logs_modify_key/fluent-bit.yaml
==63737== Parent PID: 63735
==63737== 
==63737== 
==63737== HEAP SUMMARY:
==63737==     in use at exit: 0 bytes in 0 blocks
==63737==   total heap usage: 2,399 allocs, 2,399 frees, 3,779,081 bytes allocated
==63737== 
==63737== All heap blocks were freed -- no leaks are possible
==63737== 
==63737== For lists of detected and suppressed errors, rerun with: -s
==63737== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==63738== Memcheck, a memory error detector
==63738== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63738== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63738== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/logs_modify_metadata/fluent-bit.yaml
==63738== Parent PID: 63735
==63738== 
==63738== 
==63738== HEAP SUMMARY:
==63738==     in use at exit: 0 bytes in 0 blocks
==63738==   total heap usage: 2,267 allocs, 2,267 frees, 3,775,431 bytes allocated
==63738== 
==63738== All heap blocks were freed -- no leaks are possible
==63738== 
==63738== For lists of detected and suppressed errors, rerun with: -s
==63738== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==63739== Memcheck, a memory error detector
==63739== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63739== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63739== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/logs_split_drop/fluent-bit.yaml
==63739== Parent PID: 63735
==63739== 
==63739== 
==63739== HEAP SUMMARY:
==63739==     in use at exit: 0 bytes in 0 blocks
==63739==   total heap usage: 2,236 allocs, 2,236 frees, 2,962,169 bytes allocated
==63739== 
==63739== All heap blocks were freed -- no leaks are possible
==63739== 
==63739== For lists of detected and suppressed errors, rerun with: -s
==63739== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==63740== Memcheck, a memory error detector
==63740== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63740== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63740== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/metrics_add_remove_sample_label/fluent-bit.yaml
==63740== Parent PID: 63735
==63740== 
==63740== 
==63740== HEAP SUMMARY:
==63740==     in use at exit: 0 bytes in 0 blocks
==63740==   total heap usage: 3,785 allocs, 3,785 frees, 3,400,032 bytes allocated
==63740== 
==63740== All heap blocks were freed -- no leaks are possible
==63740== 
==63740== For lists of detected and suppressed errors, rerun with: -s
==63740== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==63742== Memcheck, a memory error detector
==63742== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63742== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63742== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/metrics_modify_name_output/fluent-bit.yaml
==63742== Parent PID: 63735
==63742== 
==63742== 
==63742== HEAP SUMMARY:
==63742==     in use at exit: 0 bytes in 0 blocks
==63742==   total heap usage: 4,114 allocs, 4,114 frees, 3,023,612 bytes allocated
==63742== 
==63742== All heap blocks were freed -- no leaks are possible
==63742== 
==63742== For lists of detected and suppressed errors, rerun with: -s
==63742== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==63741== Memcheck, a memory error detector
==63741== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63741== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63741== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/metrics_modify_name/fluent-bit.yaml
==63741== Parent PID: 63735
==63741== 
==63741== 
==63741== HEAP SUMMARY:
==63741==     in use at exit: 0 bytes in 0 blocks
==63741==   total heap usage: 3,695 allocs, 3,695 frees, 3,397,224 bytes allocated
==63741== 
==63741== All heap blocks were freed -- no leaks are possible
==63741== 
==63741== For lists of detected and suppressed errors, rerun with: -s
==63741== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==63744== Memcheck, a memory error detector
==63744== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63744== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63744== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/traces_modify_metadata_output/fluent-bit.yaml
==63744== Parent PID: 63735
==63744== 
==63744== 
==63744== HEAP SUMMARY:
==63744==     in use at exit: 0 bytes in 0 blocks
==63744==   total heap usage: 5,434 allocs, 5,434 frees, 4,065,391 bytes allocated
==63744== 
==63744== All heap blocks were freed -- no leaks are possible
==63744== 
==63744== For lists of detected and suppressed errors, rerun with: -s
==63744== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==63743== Memcheck, a memory error detector
==63743== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63743== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63743== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/traces_modify_metadata/fluent-bit.yaml
==63743== Parent PID: 63735
==63743== 
==63743== 
==63743== HEAP SUMMARY:
==63743==     in use at exit: 0 bytes in 0 blocks
==63743==   total heap usage: 4,504 allocs, 4,504 frees, 3,997,171 bytes allocated
==63743== 
==63743== All heap blocks were freed -- no leaks are possible
==63743== 
==63743== For lists of detected and suppressed errors, rerun with: -s
==63743== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==63745== Memcheck, a memory error detector
==63745== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==63745== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==63745== Command: ./build/bin/fluent-bit -c examples/processor_calyptia/traces_modify_split_span/fluent-bit.yaml
==63745== Parent PID: 63735
==63745== 
==63745== 
==63745== HEAP SUMMARY:
==63745==     in use at exit: 0 bytes in 0 blocks
==63745==   total heap usage: 4,428 allocs, 4,428 frees, 3,497,717 bytes allocated
==63745== 
==63745== All heap blocks were freed -- no leaks are possible
==63745== 
==63745== For lists of detected and suppressed errors, rerun with: -s
==63745== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.