Skip to content

Commit a6fa3fb

Browse files
committed
Condensing parsers
Signed-off-by: Lynette Miles <[email protected]>
1 parent 137d485 commit a6fa3fb

File tree

3 files changed

+38
-9
lines changed

3 files changed

+38
-9
lines changed

concepts/data-pipeline.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,16 @@ For more details about the Filters available and their usage, see [Filters](../p
1313

1414
## Inputs
1515

16-
Fluent Bit provides [input plugins](../pipeline/inputs.md) to gather information from different sources. Some plugins collect data from log files, while others can gather metrics information from the operating system. There are many plugins to suit different needs.
16+
Fluent Bit provides [input plugins](../pipeline/inputs.md) to gather information from different sources. Some plugins collect data from log
17+
files, while others can gather metrics information from the operating system. There
18+
are many plugins to suit different needs.
19+
20+
## Outputs
21+
22+
[Outputs](../pipeline/outputs.md) let you define destinations for your data. Common
23+
destinations are remote services, local file systems, or other standard interfaces.
24+
Outputs are implemented as plugins.
25+
26+
## Parsers
27+
28+
Parser convert unstructured data to structured data. Use a parser to set a structure to the incoming data by using input plugins as data is collected.

pipeline/outputs.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Outputs
2+
3+
Outputs let you define destinations for your data. Common destinations are remote services, local file systems, or other standard interfaces. Outputs are implemented as plugins.
4+
5+
```mermaid
6+
graph LR
7+
accTitle: Fluent Bit data pipeline
8+
accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
9+
A[Input] --> B[Parser]
10+
B --> C[Filter]
11+
C --> D[Buffer]
12+
D --> E((Routing))
13+
E --> F[Output 1]
14+
E --> G[Output 2]
15+
E --> H[Output 3]
16+
style F stroke:darkred,stroke-width:2px;
17+
style G stroke:darkred,stroke-width:2px;
18+
style H stroke:darkred,stroke-width:2px;
19+
```
20+
21+
When an output plugin is loaded, an internal _instance_ is created. Every instance has its own independent configuration. Configuration keys are often called _properties_.

concepts/data-pipeline/parser.md renamed to pipeline/parsers.md

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,8 @@
1-
---
2-
description: Convert unstructured messages to structured messages
3-
---
1+
# Parsers
42

5-
# Parser
3+
Dealing with raw strings or unstructured messages is difficult. Having a structure makes data more usable. Set a structure to the incoming data by using input plugins as data is collected.
64

7-
Dealing with raw strings or unstructured messages is difficult. Having a structure makes data more usable. Set a structure to the incoming data by using input plugins as data is collected:
5+
Parsers are fully configurable and are independently and optionally handled by each input plugin.
86

97
```mermaid
108
graph LR
@@ -26,7 +24,7 @@ The parser converts unstructured data to structured data. As an example, conside
2624
192.168.2.20 - - [28/Jul/2006:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395
2725
```
2826

29-
This log line is a raw string without format. Structuring the log makes it easier to process the data later. If the [regular expression parser](../../pipeline/parsers/regular-expression.md) is used, the log entry could be converted to:
27+
This log line is a raw string without format. Structuring the log makes it easier to process the data later. If the [regular expression parser](./parsers/regular-expression.md) is used, the log entry could be converted to:
3028

3129
```javascript
3230
{
@@ -40,5 +38,3 @@ This log line is a raw string without format. Structuring the log makes it easie
4038
"agent": ""
4139
}
4240
```
43-
44-
Parsers are fully configurable and are independently and optionally handled by each input plugin. For more details, see [Parsers](https://docs.fluentbit.io/manual/pipeline/parsers).

0 commit comments

Comments
 (0)