Skip to content

Updated log to metrics filter configuration options table to support v4.1. Fixes #2092. #2124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Oct 30, 2025
Merged

Updated log to metrics filter configuration options table to support v4.1. Fixes #2092. #2124

Changes from 15 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
5f4a90c
Removing a readme file this is not standardized in classic-mode direc…
eschabell Oct 22, 2025
ca437dc
Updated to not use readme file in classic-config directory, but the c…
eschabell Oct 22, 2025
5bbc956
Updated to better sort some sections and update for missing entries. …
eschabell Oct 22, 2025
b630391
Merge branch 'fluent:master' into master
eschabell Oct 23, 2025
e4ea1cc
Updated configuration variables for opentelemetry output plugin, many…
eschabell Oct 23, 2025
f8ff78d
Updated configuration variables for opentelemetry output plugin, many…
eschabell Oct 23, 2025
8b007c0
Fixes reqeusted by reviewers for options table for opentelemetry outp…
eschabell Oct 24, 2025
8f2b2a1
Merge branch 'fluent:master' into master
eschabell Oct 27, 2025
81143f7
Merge branch 'fluent:master' into master
eschabell Oct 27, 2025
f93136a
Merge branch 'fluent:master' into master
eschabell Oct 28, 2025
309ea0a
Updated TLS doc configuration options table to match v4.1 release. Fi…
eschabell Oct 28, 2025
9a3b32b
Merge branch 'fluent:master' into master
eschabell Oct 28, 2025
c7a4b2b
Updated log to metrics filter configuration options table to support …
eschabell Oct 28, 2025
e1b9032
Fixes to review comments, removed output references and corrected buc…
eschabell Oct 29, 2025
e08751f
Merge branch 'master' into erics_log_metrics_filter_updates
eschabell Oct 29, 2025
d334e1c
Update pipeline/filters/log_to_metrics.md
eschabell Oct 30, 2025
e36bebf
Update pipeline/filters/log_to_metrics.md
eschabell Oct 30, 2025
6f2f3dd
Update pipeline/filters/log_to_metrics.md
eschabell Oct 30, 2025
503d4ec
Update pipeline/filters/log_to_metrics.md
eschabell Oct 30, 2025
647fac0
Update pipeline/filters/log_to_metrics.md
eschabell Oct 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 29 additions & 18 deletions pipeline/filters/log_to_metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@

# Logs to metrics

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=768830f6-8d2d-4231-9e5e-259ce6797ba5" />
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=768830f6-8d2d-4231-9e5e-259ce6797ba5" alt="tracking"/>

The _log to metrics_ filter lets you generate log-derived metrics. It supports modes to count records, provide a gauge for field values, or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values.

This filter doesn't actually act as a record filter and therefore doesn't change or drop records. All records will pass through this filter untouched, and any generated metrics will be emitted into a separate metric pipeline.
This filter doesn't actually act as a record filter and therefore does not change or drop records. All records will pass through this filter untouched, and any generated metrics will be emitted into a separate metric pipeline.

Check warning on line 11 in pipeline/filters/log_to_metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Contractions] Feel free to use 'doesn't' instead of 'does not'.

Raw Output:
{"message": "[FluentBit.Contractions] Feel free to use 'doesn't' instead of 'does not'.", "location": {"path": "pipeline/filters/log_to_metrics.md", "range": {"start": {"line": 11, "column": 67}}}, "severity": "INFO"}

{% hint style="warning" %}

Expand All @@ -20,21 +20,32 @@

The plugin supports the following configuration parameters:

| Parameter | Description | Value format |
|---|---|---|
| `tag` | Required. Defines the tag for the generated metrics record. | |
| `metric_mode` | Required. Defines the mode for the metric. Valid values are `counter`, `gauge` or `histogram`. | |
| `metric_name` | Required. Sets the name of the metric. | |
| `metric_description` | Required. Sets a description for the metric. | |
| `bucket` | Required for mode `histogram`. Defines a bucket for histograms. | For example, `0.75` |
| `add_label` | Adds a custom label `NAME` and set the value to the value of `KEY`. | |
| `label_field` | Includes a record field as label dimension in the metric. | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. |
| `value_field` | Required for modes `gauge` and `histogram`. Specifies the record field that holds a numerical value. | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. |
| `kubernetes_mode` | If enabled, adds `pod_id`, `pod_name`, `namespace_name`, `docker_id` and `container_name` to the metric as labels. This option is intended to be used in combination with the [Kubernetes](./kubernetes.md) filter plugin, which fills those fields. | |
| `Regex` | Includes records in which the content of `KEY` matches the regular expression. | `KEY REGEX` |
| `Exclude` | Excludes records in which the content of `KEY` matches the regular expression. | `KEY REGEX` |
| `Flush_Interval_Sec` | The interval for metrics emission, in seconds. If `Flush_Interval_Sec` and `Flush_Interval_Nsec` are either both unset or both set to `0`, the filter emits metrics immediately after each filter match. Otherwise, if either parameter is set to a non-zero value, the filter emits metrics at the specified interval. Longer intervals help lower resource consumption in high-load situations. Default value: `0`. | |
| `Flush_Interval_Nsec` | The interval for metrics emission, in nanoseconds. This parameter works in conjunction with `Flush_Interval_Sec`. Default value: `0`. | |
| Key | Description | Default | Format |
|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| `add_label` | Adds a custom label `NAME` and set the value to the value of `KEY`. | _none_ | `NAME KEY` |
| `alias` | Sets an alias, useful for multiple instances of the same filter plugin. | _none_ | |

Check warning on line 26 in pipeline/filters/log_to_metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "useful" that imply ease of use, because the user may find this action difficult.

Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like \"useful\" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/filters/log_to_metrics.md", "range": {"start": {"line": 26, "column": 44}}}, "severity": "WARNING"}
| `bucket` | Optional for `metric_mode` `histogram`. If not set, default Prometheus-style buckets are used. | _none_ | For example, `0.75` |
| `discard_logs` | Flag that defines if logs should be discarded after processing. This applies for all logs, no matter if they have emitted metrics or not. | `false` | |
| `emitter_name` | Name of the emitter (advanced users). | _none_ | |
| `emitter_mem_buf_limit` | set a buffer limit to restrict memory usage of metrics emitter. | `10M` | |
| `exclude` | Excludes records in which the content of `KEY` matches the regular expression `REGEX`. | _none_ | `KEY REGEX` |
| `flush_interval_sec` | The interval for metrics emission, in seconds. If `flush_interval_sec` and `flush_interval_nsec` are either both unset or both set to `0`, the filter emits metrics immediately after each filter match. Otherwise, if either parameter is set to a non-zero value, the filter emits metrics at the specified interval. Longer intervals help lower resource consumption in high-load situations. | `0` | |
| `flush_interval_nsec` | The interval for metrics emission, in nanoseconds. This parameter works in conjunction with `flush_interval_sec`. | `0` | |
| `kubernetes_mode` | If enabled, adds `pod_id`, `pod_name`, `namespace_name`, `docker_id` and `container_name` to the metric as labels. This option is intended to be used in combination with the [Kubernetes](./kubernetes.md) filter plugin, which fills those fields. | `false` | |
| `label_field` | Includes a record field as label dimension in the metric. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. |
| `log_level` | Specifies the log level for filter plugin. If not set here, plugin uses global log level in `service` section. | `info` | |
| `log_supress_interval` | Suppresses log messages from filter plugin that appear similar within a specified time interval. `0` no suppression. | `0` | |
| `match` | Set a tag pattern to match records that filter should process. Exact matches or wildcards. | _none_ | |
| `match_regex` | Set a regular expression to match tags for filter routing. This allows more flexible matching compared to simple wildcards. | _none_ | |

Check warning on line 39 in pipeline/filters/log_to_metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "simple" that imply ease of use, because the user may find this action difficult.

Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like \"simple\" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/filters/log_to_metrics.md", "range": {"start": {"line": 39, "column": 135}}}, "severity": "WARNING"}
| `metric_description` | Sets a description for the metric. | _none_ | |
| `metric_mode` | Defines the mode for the metric. Valid values are `counter`, `gauge` or `histogram`. | `counter` | |
| `metric_name` | Sets the name of the metric. | `a` | |
| `metric_namespace` | Sets the namespace of the metric. | `log_metric` | |
| `metric_subsystem` | Subsystem of the metric. | _none_ | |
| `regex` | Includes records in which the content of `KEY` matches the regular expression `REGEX`. | _none_ | `KEY REGEX` |
| `tag` | Defines the tag for the generated metrics record. | _none_ | |
| `value_field` | Required for modes `gauge` and `histogram`. Specifies the record field that holds a numerical value. | _none_ | Name of record key. Supports [record accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) notation for nested fields. |


## Examples

Expand Down Expand Up @@ -532,4 +543,4 @@

{% endhint %}

This filter also attaches Kubernetes labels to each metric, identical to the behavior of `label_field`. This results in two sets for the histogram.
This filter also attaches Kubernetes labels to each metric, identical to the behavior of `label_field`. This results in two sets for the histogram.