Add integration tests

[lamr02n]

Summary

To make sure that the first three pipeline stages work as intended, we can set up an integration test.

Basic example

For this, we could have two files, one for input and one for output, where the input file contains a large number of log lines and the output file contains the ones, we manually filtered out (by doing manually what the pipeline should do). Then, we could add a docker container for retrieving the resulting log lines from the Prefilter and write them to a file. At the end, we can compare the files and see if the pipeline got the same result as we calculated. All this could be set up with GitHub Actions.

Motivation

This can serve as an additional test regarding the data preparation phases. Currently, it can happen that changes made to any of the modules lead to bugs we can only see with manual testing. Automated testing would reduce that work and ensure that the first half of the pipeline works as it should.

[stefanDeveloper]

Can the benchmarking tests be considered integration tests, or is this issues still open?