Plugins not aware of data format being passed to them

[MKLeb]

At the moment, processors and forwarders have no knowledge of the data being passed to them by the previous stage of the pipeline.

For instance, if we have a pipeline with a log collector, no processor, and a disk forwarder, the disk forwarder has no way of filtering pieces of the collected log event to dump to disk. Currently, the logs collector passes a dictionary in the data field with two keys: log_line and template.

Let's say we only want to dump the template to disk. Currently, this is not possible. We need a way to filter, add, remove, etc... fields from a collected event before passing it on to the next stage in the pipeline.

[MKLeb]

The current proposed solutions (by @s0undt3ch and @MKLeb) are as follows:

• Create a processors plugin (or collection of plugins) for the addition, removal, filtering, altering, etc... of the fields of a collected event. This is likely the most thorough solution, as it allows for extensibility beyond this one use case.
• Create one plugin that takes a collected event, and formats it into another one (the layout of which is configured with YAML). This is equally as thorough as the above proposal, but not as scalable or extensible.
• Create a processor that allows the calling of a salt module (using the already loaded __salt__), and then have users implement their own processors in the form of exec modules under the _modules directory within salt itself. This allows for complete control over the processing and formatting of the event.
• Let users implement these as custom SAF plugins when the use case arises. This is the most customized way, but requires a lot of work on the part of the user.