Merge pull request #2040 from fluent/lynettemiles/sc-144419/fluent-clean-up-concepts-section

Author: esmerel

SUMMARY.md

@@ -14,13 +14,7 @@
 
 * [Key concepts](concepts/key-concepts.md)
 * [Buffering](concepts/buffering.md)
-* [Data pipeline](concepts/data-pipeline/README.md)
-  * [Input](concepts/data-pipeline/input.md)
-  * [Parser](concepts/data-pipeline/parser.md)
-  * [Filter](concepts/data-pipeline/filter.md)
-  * [Buffer](concepts/data-pipeline/buffer.md)
-  * [Router](concepts/data-pipeline/router.md)
-  * [Output](concepts/data-pipeline/output.md)
+* [Data pipeline](concepts/data-pipeline.md)
 
 ## Installation
 
@@ -91,7 +85,7 @@
 ## Data pipeline
 
 * [Pipeline monitoring](pipeline/pipeline-monitoring.md)
-* [Inputs](pipeline/inputs/README.md)
+* [Inputs](pipeline/inputs.md)
   * [Collectd](pipeline/inputs/collectd.md)
   * [CPU metrics](pipeline/inputs/cpu-metrics.md)
   * [Disk I/O metrics](pipeline/inputs/disk-io-metrics.md)
@@ -136,7 +130,7 @@
   * [Windows Event logs (winevtlog)](pipeline/inputs/windows-event-log-winevtlog.md)
   * [Windows Event logs (winlog)](pipeline/inputs/windows-event-log.md)
   * [Windows exporter metrics](pipeline/inputs/windows-exporter-metrics.md)
-* [Parsers](pipeline/parsers/README.md)
+* [Parsers](pipeline/parsers.md)
   * [Configuring parsers](pipeline/parsers/configuring-parser.md)
   * [JSON](pipeline/parsers/json.md)
   * [Regular expression](pipeline/parsers/regular-expression.md)
@@ -152,7 +146,7 @@
   * [SQL](pipeline/processors/sql.md)
   * [Filters as processors](pipeline/processors/filters.md)
   * [Conditional processing](pipeline/processors/conditional-processing.md)
-* [Filters](pipeline/filters/README.md)
+* [Filters](pipeline/filters.md)
   * [AWS metadata](pipeline/filters/aws-metadata.md)
   * [CheckList](pipeline/filters/checklist.md)
   * [ECS metadata](pipeline/filters/ecs-metadata.md)
@@ -175,7 +169,8 @@
   * [Throttle](pipeline/filters/throttle.md)
   * [Type converter](pipeline/filters/type-converter.md)
   * [Wasm](pipeline/filters/wasm.md)
-* [Outputs](pipeline/outputs/README.md)
+* [Router](pipeline/router.md)
+* [Outputs](pipeline/outputs.md)
   * [Amazon CloudWatch](pipeline/outputs/cloudwatch.md)
   * [Amazon Kinesis Data Firehose](pipeline/outputs/firehose.md)
   * [Amazon Kinesis Data Streams](pipeline/outputs/kinesis.md)

concepts/buffering.md

@@ -8,10 +8,30 @@ When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memor
 
 Buffering is the ability to store the records, and continue storing incoming data while previous data is processed and delivered. Buffering in memory is the fastest mechanism, but there are scenarios requiring special strategies to deal with [backpressure](../administration/backpressure.md), data safety, or to reduce memory consumption by the service in constrained environments.
 
+```mermaid
+graph LR
+    accTitle: Fluent Bit data pipeline
+    accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
+    A[Input] --> B[Parser]
+    B --> C[Filter]
+    C --> D[Buffer]
+    D --> E((Routing))
+    E --> F[Output 1]
+    E --> G[Output 2]
+    E --> H[Output 3]
+    style D stroke:darkred,stroke-width:2px;
+```
+
 Network failures or latency in third party service is common. When data can't be delivered fast enough and new data to process arrives, the system can face backpressure.
 
 Fluent Bit buffering strategies are designed to solve problems associated with backpressure and general delivery failures. Fluent Bit offers a primary buffering mechanism in memory and an optional secondary one using the file system. With this hybrid solution you can accommodate any use case safely and keep a high performance while processing your data.
 
 These mechanisms aren't mutually exclusive. When data is ready to be processed or delivered it's always be in memory. Other data in the queue might be in the file system until is ready to be processed and moved up to memory.
 
+The `buffer` phase contains the data in an immutable state, meaning that no other filter can be applied.
+
+Buffered data uses the Fluent Bit internal binary representation, which isn't raw text.
+
+To avoid data loss in case of system failures, Fluent Bit offers a buffering mechanism in the file system that acts as a backup system.
+
 To learn more about the buffering configuration in Fluent Bit, see [Buffering and Storage](../administration/buffering-and-storage.md).

concepts/data-pipeline.md

@@ -0,0 +1,27 @@
+# Data pipeline
+
+The Fluent Bit data pipeline incorporates several specific concepts. Data processing flows through the pipeline following these concepts in order.
+
+## Filters
+
+[Filters](../pipeline/filters.md) let you alter the collected data before delivering it to a destination. In production environments you need full control of the data you're collecting. Using filters lets you control data before processing.
+
+## Buffer
+
+The [`buffer`](./buffering.md) phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.
+
+## Inputs
+
+Fluent Bit provides [input plugins](../pipeline/inputs.md) to gather information from different sources. Some plugins collect data from log files, and others gather metrics information from the operating system. There are many plugins to suit different needs.
+
+## Outputs
+
+[Output plugins](../pipeline/outputs.md) let you define destinations for your data. Common destinations are remote services, local file systems, or other standard interfaces.
+
+## Parsers
+
+[Parsers](../pipeline/parsers.md) convert unstructured data to structured data. Use a parser to set a structure to the incoming data by using input plugins as data is collected.
+
+## Route
+
+[Routing](../pipeline/router.md) is a core feature that lets you route your data through filters, and then to one or multiple destinations. The router relies on the concept of [tags](./key-concepts.md#tag) and [matching](./key-concepts.md#match) rules.

concepts/data-pipeline/README.md

@@ -58,7 +58,7 @@ to represent events. This format is still supported for reading input event stre
 
 ## Filtering
 
-You might need to perform modifications on an Event's content. The process to alter, append to, or drop Events is called [_filtering_](data-pipeline/filter.md).
+You might need to perform modifications on an event's content. The process to alter, append to, or drop Events is called [_filtering_](../pipeline/filters.md).
 
 Use filtering to:
 
@@ -68,15 +68,15 @@ Use filtering to:
 
 ## Tag
 
-Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string used in a later stage by the Router to decide which Filter or [Output](data-pipeline/output.md) phase it must go through.
+Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string used in a later stage by the Router to decide which Filter or [Output](../pipeline/outputs.md) phase it must go through.
 
-Most tags are assigned manually in the configuration. If a tag isn't specified, Fluent Bit assigns the name of the [Input](data-pipeline/input.md) plugin instance where that Event was generated from.
+Most tags are assigned manually in the configuration. If a tag isn't specified, Fluent Bit assigns the name of the [Input](../pipeline/inputs.md) plugin instance where that Event was generated from.
 
 {% hint style="info" %}
 The [Forward](../pipeline/inputs/forward.md) input plugin doesn't assign tags. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client.
 {% endhint %}
 
-A tagged record must always have a Matching rule. To learn more about Tags and Matches, see [Routing](data-pipeline/router.md).
+A tagged record must always have a Matching rule. To learn more about Tags and Matches, see [Routing](../pipeline/router.md).
 
 ## Timestamp
 
@@ -97,7 +97,7 @@ where:
 
 Fluent Bit lets you route your collected and processed Events to one or multiple destinations. A _Match_ represents a rule to select Events where a Tag matches a defined rule.
 
-To learn more about Tags and Matches, see [Routing](data-pipeline/router.md).
+To learn more about Tags and Matches, see [Routing](../pipeline/router.md).
 
 ## Structured messages
 

concepts/data-pipeline/buffer.md

@@ -12,7 +12,7 @@ The following steps assume you are familiar with configuring Fluent Bit using te
 
 #### Configuration directory
 
-In your file system, prepare a specific directory that will be used as an entry point for the build system to lookup and parse the configuration files. This directory must contain a minimum of one configuration file called `fluent-bit.conf` containing the required [`SERVICE`](/administration/configuring-fluent-bit/yaml/service-section.md), [`INPUT`](/concepts/data-pipeline/input.md), and [`OUTPUT`](/concepts/data-pipeline/output.md) sections.
+In your file system, prepare a specific directory that will be used as an entry point for the build system to lookup and parse the configuration files. This directory must contain a minimum of one configuration file, called `fluent-bit.conf`, that contains the required [`SERVICE`](/administration/configuring-fluent-bit/yaml/service-section.md), [`INPUT`](../../pipeline/input.md), and [`OUTPUT`](../../pipeline/output.md) sections.
 
 As an example, create a new `fluent-bit.yaml` file or `fluent-bit.conf` file:
 
@@ -84,4 +84,4 @@ $ bin/fluent-bit
 
 ...
 [0] cpu.local: [1539984752.000347547, {"cpu_p"=>0.750000, "user_p"=>0.500000, "system_p"=>0.250000, "cpu0.p_cpu"=>1.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>0.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000, "cpu2.p_cpu"=>0.000000, "cpu2.p_user"=>0.000000, "cpu2.p_system"=>0.000000, "cpu3.p_cpu"=>1.000000, "cpu3.p_user"=>1.000000, "cpu3.p_system"=>0.000000}]
-```
+```

concepts/key-concepts.md

@@ -1,8 +1,4 @@
----
-description: Modify, enrich or drop your records
----
-
-# Filter
+# Filters
 
 In production environments you need full control of the data you're collecting. Filtering lets you alter the collected data before delivering it to a destination.
 
@@ -25,5 +21,3 @@ Filtering is implemented through plugins. Each available filter can be used to m
 Fluent Bit support many filters. A common use case for filtering is Kubernetes deployments. Every pod log needs the proper metadata associated with it.
 
 Like input plugins, filters run in an instance context, which has its own independent configuration. Configuration keys are often called _properties_.
-
-For more details about the Filters available and their usage, see [Filters](https://docs.fluentbit.io/manual/pipeline/filters).

installation/sources/build-with-static-configuration.md

@@ -4,9 +4,9 @@ description: Powerful and flexible routing
 
 # Rewrite tag
 
-Tags make [routing](../../concepts/data-pipeline/router.md) possible. Tags are set in the configuration of the `INPUT` definitions where the records are generated. There are scenarios where you might want to modify the tag in the pipeline to perform more advanced and flexible routing.
+Tags make [routing](../../pipeline/router.md) possible. Tags are set in the configuration of the `INPUT` definitions where the records are generated. There are scenarios when you might want to modify the tag in the pipeline to perform more advanced and flexible routing.
 
-The _Rewrite Tag_ filter lets you re-emit a record under a new tag. Once a record has been re-emitted, the original record can be preserved or discarded.
+The _Rewrite Tag_ filter lets you re-emit a record under a new tag. After a record is re-emitted, the original record can be preserved or discarded.
 
 The Rewrite Tag filter defines rules that match specific record key content against a regular expression. If a match exists, a new record with the defined tag will be emitted, entering from the beginning of the pipeline. Multiple rules can be specified and are processed in order until one of them matches.