-
Notifications
You must be signed in to change notification settings - Fork 540
fix confusing/misleading info about parser support in YAML config #2133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 4 commits
4a50687
85532d4
22fdbc2
40ba1e9
439558c
11abb70
0071503
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,22 +1,21 @@ | ||
| # Pipeline | ||
|
|
||
| The `pipeline` section defines the flow of how data is collected, processed, and sent to its final destination. It encompasses the following core concepts: | ||
| The `pipeline` section defines the flow of how data is collected, processed, and sent to its final destination. This section contains the following subsections: | ||
|
|
||
| | Name | Description | | ||
| | ---- | ----------- | | ||
| | `inputs` | Specifies the name of the plugin responsible for collecting or receiving data. This component serves as the data source in the pipeline. Examples of input plugins include `tail`, `http`, and `random`. | | ||
| | `processors` | **Unique to YAML configuration**, processors are specialized plugins that handle data processing directly attached to input plugins. Unlike filters, processors aren't dependent on tag or matching rules. Instead, they work closely with the input to modify or enrich the data before it reaches the filtering or output stages. Processors are defined within an input plugin section. | | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure I follow the reasoning around removing this. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it's part of a larger change that I'm planning to make in the near future, but TLDR is that I think it's misleading to include processors on the same "level" as inputs/filters/outputs since processors are nested one level below them; this table layout makes them all appear to be on the same hierarchal level. when I revisit this page (and others) I'm hoping to make it clearer exactly how processors get attached to inputs and outputs, and that that they're not a direct parallel to filters. |
||
| | `filters` | Filters are used to transform, enrich, or discard events based on specific criteria. They allow matching tags using strings or regular expressions, providing a more flexible way to manipulate data. Filters run as part of the main event loop and can be applied across multiple inputs and filters. Examples of filters include `modify`, `grep`, and `nest`. | | ||
| | `outputs` | Defines the destination for processed data. Outputs specify where the data will be sent, such as to a remote server, a file, or another service. Each output plugin is configured with matching rules to determine which events are sent to that destination. Common output plugins include `stdout`, `elasticsearch`, and `kafka`. | | ||
|
|
||
| ## Example configuration | ||
|
|
||
| {% hint style="info" %} | ||
|
|
||
| **Note:** Processors can be enabled only by using the YAML configuration format. Classic mode configuration format doesn't support processors. | ||
| Unlike filters, processors and parsers aren't defined within a unified section of YAML configuration files and don't use tag matching. Instead, each input or output defined in the configuration file can have a `parsers` key and `processors` key to configure the parsers and processors for that specific plugin. | ||
|
|
||
| {% endhint %} | ||
|
|
||
| ## Example configuration | ||
|
|
||
| Here's an example of a pipeline configuration: | ||
|
|
||
| {% tabs %} | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.