Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion about/sandbox-and-lab-resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

# Sandbox and lab resources

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=83d7b072-bdfc-4f7d-810d-3ab42267ff54" />

## Open Source labs - environment required

The following are open source labs where you will need to spin up resources to run through the lab in details
Expand All @@ -32,4 +34,4 @@

This workshop by Amazon goes through common Kubernetes logging patterns and routing data to OpenSearch and visualizing with OpenSearch dashboards

{% embed url="https://eksworkshop.com/" %}
{% embed url="https://eksworkshop.com/" %}

Check warning on line 37 in about/sandbox-and-lab-resources.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [FluentBit.WordList] Use 'URL' instead of 'url'. Raw Output: {"message": "[FluentBit.WordList] Use 'URL' instead of 'url'.", "location": {"path": "about/sandbox-and-lab-resources.md", "range": {"start": {"line": 37, "column": 10}}}, "severity": "INFO"}
4 changes: 3 additions & 1 deletion administration/backpressure.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Backpressure

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=63e37cfe-9ce3-4a18-933a-76b9198958c1" />

It's possible for logs or data to be ingested or created faster than the ability to flush it to some destinations. A common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates _backpressure_, leading to high memory consumption in the service.

To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data an input plugin can ingest. Restriction is done through the configuration parameters `Mem_Buf_Limit` and `storage.Max_Chunks_Up`.
Expand Down Expand Up @@ -68,4 +70,4 @@ With `storage.type filesystem` and `storage.max_chunks_up`, the following log me
```text
[input] {input name or alias} paused (storage buf overlimit)
[input] {input name or alias} resume (storage buf overlimit)
```
```
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Configuration file

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=5e67142e-3887-4b56-b940-18494bcc23a7" />

One of the ways to configure Fluent Bit is using a main configuration file. Fluent Bit allows the use one configuration file that works at a global scope and uses the defined [Format and Schema](format-schema.md).

The main configuration file supports four sections:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Variables

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=1731c7b5-34c6-424f-bfc6-88c2aa71e81f" />

Fluent Bit supports the usage of environment variables in any value associated to a key when using a configuration file.

The variables are case sensitive and can be used in the following format:
Expand Down
2 changes: 2 additions & 0 deletions administration/configuring-fluent-bit/multiline-parsing.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Multiline parsing

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=e19a4c14-a9e4-4163-8f3a-52196eb9a585" />

In an ideal world, applications might log their messages within a single line, but in reality applications generate multiple log messages that sometimes belong to the same context. Processing this information can be complex, like in application stack traces, which always have multiple log lines.

Fluent Bit v1.8 implemented a unified Multiline core capability to solve corner cases.
Expand Down
4 changes: 3 additions & 1 deletion administration/configuring-fluent-bit/yaml.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# YAML configuration

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=864c6f0e-8977-4838-8772-84416943548e" />

## Before you get started

Fluent Bit traditionally offered a `classic` configuration mode, a custom configuration format that's phasing out. While `classic` mode has served well for many years, it has several limitations. Its basic design only supports grouping sections with key-value pairs and lacks the ability to handle sub-sections or complex data structures like lists.
Expand Down Expand Up @@ -41,4 +43,4 @@ To access detailed configuration guides for each section, use the following link
- [Environment Variables Section documentation](./yaml/environment-variables-section.md)
- Information on setting environment variables and their scope within Fluent Bit.
- [Includes Section documentation](./yaml/includes-section.md)
- Description on how to include external YAML files.
- Description on how to include external YAML files.
4 changes: 3 additions & 1 deletion administration/memory-management.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Memory management

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=5cc3ce54-e910-4ebf-85f5-f02530b3e11b" />

You might need to estimate how much memory Fluent Bit could be using in scenarios like containerized environments where memory limits are essential.

To make an estimate, in-use input plugins must set the `Mem_Buf_Limit`option. Learn more about it in [Backpressure](backpressure.md).
Expand Down Expand Up @@ -33,4 +35,4 @@ FLB_HAVE_PROXY_GO FLB_HAVE_JEMALLOC JEMALLOC_MANGLE FLB_HAVE_REGEX
FLB_HAVE_C_TLS FLB_HAVE_SETJMP FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY
```

If the `FLB_HAVE_JEMALLOC` option is listed in `Build Flags`, jemalloc is enabled.
If the `FLB_HAVE_JEMALLOC` option is listed in `Build Flags`, jemalloc is enabled.
4 changes: 3 additions & 1 deletion pipeline/filters/log_to_metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ description: Generate metrics from logs

# Logs to metrics

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=768830f6-8d2d-4231-9e5e-259ce6797ba5" />

The _log to metrics_ filter lets you generate log-derived metrics. It supports modes to count records, provide a gauge for field values, or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values.

This filter doesn't actually act as a record filter and therefore doesn't change or drop records. All records will pass through this filter untouched, and any generated metrics will be emitted into a separate metric pipeline.
Expand Down Expand Up @@ -530,4 +532,4 @@ The `+Inf` bucket will always be included regardless of the buckets you specify.

{% endhint %}

This filter also attaches Kubernetes labels to each metric, identical to the behavior of `label_field`. This results in two sets for the histogram.
This filter also attaches Kubernetes labels to each metric, identical to the behavior of `label_field`. This results in two sets for the histogram.
4 changes: 3 additions & 1 deletion pipeline/filters/lua.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Lua

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=f519378e-536c-4b25-8949-ee6ed8d8d6c1" />

The _Lua_ filter lets you modify incoming records (or split one record into multiple records) using custom [Lua](https://www.lua.org/) scripts.

A Lua-based filter requires two steps:
Expand Down Expand Up @@ -804,4 +806,4 @@ pipeline:

```text
test: [[1731990257.781970977, {}], {"my_env"=>{"A"=>"aaa", "C"=>"ccc", "HOSTNAME"=>"monox-2.lan", "B"=>"bbb"}, "rand_value"=>4805047635809401856}]
```
```
4 changes: 3 additions & 1 deletion pipeline/filters/type-converter.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Type converter

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=8984f540-d95a-462b-8a08-09f72f5fab63" />

The _Type converter_ filter plugin converts data types and appends new key-value pairs.

You can use this filter in combination with plugins which expect incoming string value. For example, [Grep](grep.md) and [Modify](modify.md).
Expand Down Expand Up @@ -88,4 +90,4 @@ The output will be

```text
[0] mem.0: [1639915154.160159749, {"Mem.total"=>8146052, "Mem.used"=>4513564, "Mem.free"=>3632488, "Swap.total"=>1918356, "Swap.used"=>0, "Swap.free"=>1918356, "Mem.total_str"=>"8146052", "Mem.used_str"=>"4513564", "Mem.free_str"=>"3632488"}]
```
```
4 changes: 3 additions & 1 deletion pipeline/processors/content-modifier.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Content modifier

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=ee1ad690-a3e9-434f-9635-3e53c670e96c" />

The _content modifier_ processor lets you manipulate the content, metadata, and attributes of logs and traces.

Similar to how filters work, this processor uses a unified mechanism to perform operations for data manipulation. The most significant difference is that processors perform better than filters, and when chaining them, there are no encoding/decoding performance penalties.
Expand Down Expand Up @@ -281,4 +283,4 @@ pipeline:
```

{% endtab %}
{% endtabs %}
{% endtabs %}
4 changes: 3 additions & 1 deletion pipeline/processors/labels.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Labels

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=1e9a2474-00c3-4d8d-b170-79996be7af79" />

The _labels_ processor lets you manipulate the labels of metrics.

Similar to filters, this processor presents an enriching/modifying mechanism to perform operations for labels manipulation. The most significant difference is that processors perform better than filters, and when chaining them there are no encoding or decoding performance penalties.
Expand Down Expand Up @@ -143,4 +145,4 @@ pipeline:
```

{% endtab %}
{% endtabs %}
{% endtabs %}
2 changes: 2 additions & 0 deletions pipeline/processors/metrics-selector.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Metrics selector

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=326269f3-cfea-472d-9169-1de32c142b90" />

The _metrics selector_ processor lets you choose which metrics to include or exclude, similar to the [grep](../filters/grep.md) filter for logs.

## Configuration parameters
Expand Down
4 changes: 3 additions & 1 deletion pipeline/processors/sql.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# SQL

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=6bd80893-c66f-4950-9e6d-c21358e9e8c9" />

The _SQL_ processor lets you use conditional expressions to select content from logs. This processor doesn't depend on a database or table. Instead, your queries run on the stream.

This processor differs from the stream processor interface that runs after filters.
Expand Down Expand Up @@ -82,4 +84,4 @@ The resulting output resembles the following:
"date": 1711059261.630668,
"http_domain": "fluentbit.io"
}
```
```