Skip to content

in_mem_metrics: initial documentation. #1170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

in_mem_metrics: initial documentation. #1170

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4b7e3dd
in_mem_metrics: initial documentation.
pwhelan Jul 26, 2023
234a29c
Update pipeline/inputs/mem-metrics.md
pwhelan Nov 4, 2025
816f19e
Update pipeline/inputs/mem-metrics.md
pwhelan Nov 4, 2025
c9b624b
Update pipeline/inputs/mem-metrics.md
pwhelan Nov 4, 2025
978770a
Update pipeline/inputs/mem-metrics.md
pwhelan Nov 4, 2025
656fbd3
Update pipeline/inputs/mem-metrics.md
pwhelan Nov 4, 2025
8f2ee91
Update pipeline/inputs/mem-metrics.md
pwhelan Nov 4, 2025
d4c50a5
Update pipeline/inputs/mem-metrics.md
pwhelan Nov 4, 2025
1384ad2
Update pipeline/inputs/mem-metrics.md
pwhelan Nov 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions pipeline/inputs/mem-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Memory metrics

The _mem_metrics_, or memory metrics plugin, generates metrics for memory statistics for processes using the `smaps_rollup` file for each process in /proc. This plugin works exclusively on Linux.

Check warning on line 3 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'mem_metrics'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'mem_metrics'?", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 3, "column": 6}}}, "severity": "INFO"}

# Configuration parameters
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD025/single-title/single-h1 Multiple top-level headings in the same document [Context: "# Configuration parameters"]


| **Key*** | Description | Default |
| :------------ | :------------------------------------------------- | :----------- |
| `proc_path` | The path of the proc pseudo filesystem. | `/proc` |
| `filter_exec` | Filter for a single executable by path. | **inactive** |
| `filter_cmd` | Filter by command line. | **inactive** |
| `filter_pid` | Filter by comma delimited list of PIDs. | **inactive** |

Check warning on line 12 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'PIDs'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'PIDs'?", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 12, "column": 55}}}, "severity": "INFO"}
| `interval_sec` | Set the interval seconds between events generation. | `5` |
| `interval_nsec` | Set the nanoseconds interval (sub seconds). | `0` |

The `filter_pid` can include or be set to either `0` or `self` to refer to the fluent-bit process itself.

Check warning on line 16 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.FluentBitCasing] Use the proper noun 'Fluent Bit' with correct casing and spacing.

Raw Output:
{"message": "[FluentBit.FluentBitCasing] Use the proper noun 'Fluent Bit' with correct casing and spacing.", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 16, "column": 80}}}, "severity": "WARNING"}

## Get started

You can run the plugin from the command line or through a configuration file. By default metrics will be generated for all processes the current user can analyze.

### Command line

```bash
$ fluent-bit -i mem_metrics -o stdout
Fluent Bit v2.1.8
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

2023-07-26T14:37:49.919778443Z node_smaps_rollup_rss{pid="4540"} = 21064
2023-07-26T14:37:49.919778443Z node_smaps_rollup_pss{pid="4540",type="clean"} = 9598
2023-07-26T14:37:49.919778443Z node_smaps_rollup_pss{pid="4540",type="dirty"} = 5998
2023-07-26T14:37:49.919778443Z node_smaps_rollup_pss{pid="4540",type="anon"} = 5996
2023-07-26T14:37:49.919778443Z node_smaps_rollup_pss{pid="4540",type="file"} = 3602
2023-07-26T14:37:49.919778443Z node_smaps_rollup_pss{pid="4540",type="shmem"} = 0
2023-07-26T14:37:49.919778443Z node_smaps_rollup_shared{pid="4540",type="clean"} = 15008
2023-07-26T14:37:49.919778443Z node_smaps_rollup_shared{pid="4540",type="dirty"} = 4
2023-07-26T14:37:49.919778443Z node_smaps_rollup_private{pid="4540",type="clean"} = 56
2023-07-26T14:37:49.919778443Z node_smaps_rollup_private{pid="4540",type="dirty"} = 5996
2023-07-26T14:37:49.919778443Z node_smaps_rollup_referenced{pid="4540"} = 21064
2023-07-26T14:37:49.919778443Z node_smaps_rollup_anonymous{pid="4540"} = 5996
2023-07-26T14:37:49.919778443Z node_smaps_rollup_lazy_free{pid="4540"} = 0
2023-07-26T14:37:49.919778443Z node_smaps_rollup_anon_huge_pages{pid="4540"} = 0
2023-07-26T14:37:49.919778443Z node_smaps_rollup_shmem_pmd_mapped{pid="4540"} = 0
2023-07-26T14:37:49.919778443Z node_smaps_rollup_file_pmd_mapped{pid="4540"} = 0
2023-07-26T14:37:49.919778443Z node_smaps_rollup_shared_hugetlb{pid="4540"} = 0
2023-07-26T14:37:49.919778443Z node_smaps_rollup_private_hugetlb{pid="4540"} = 0
2023-07-26T14:37:49.919778443Z node_smaps_rollup_swap{pid="4540"} = 0
2023-07-26T14:37:49.919778443Z node_smaps_rollup_swap_pss{pid="4540"} = 0
2023-07-26T14:37:49.919778443Z node_smaps_rollup_locked{pid="4540"} = 0
```

### Configuration file

Minimal pipeline for logging memory metrics to the console:

```ini
[INPUT]
Name mem_metrics
Tag mem

[OUTPUT]
Name stdout
Match mem
```

Log all instances of Fluent Bit to Prometheus:

```ini
[INPUT]
Name mem_metrics
Filter_cmd fluent-bit
Tag mem

[OUTPUT]
Name prometheus_remote_write
Match mem
Host localhost
Port 9090
Uri /prometheus/v1/write?prometheus_server=mem
```

### Exposed metrics

All metrics are logged as gauges since they can both increase and decrease. Supported gauges are:

- node_smaps_rollup_rss

Check warning on line 88 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_rss'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_rss'?", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 88, "column": 5}}}, "severity": "INFO"}
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_pss
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

Comment on lines +88 to +89
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
- node_smaps_rollup_rss
- node_smaps_rollup_pss
- node_smaps_rollup_rss
- node_smaps_rollup_pss
- type=clean
- type=dirty
- type=anon
- type=file
- type=shmem
- node_smaps_rollup_shared

- type=clean
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 2; Actual: 4]

- type=dirty
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 2; Actual: 4]

- type=anon
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 2; Actual: 4]

- type=file
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 2; Actual: 4]

- type=shmem
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 2; Actual: 4]

- node_smaps_rollup_shared

Check warning on line 95 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_shared'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_shared'?", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 95, "column": 6}}}, "severity": "INFO"}
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- type=clean
- type=dirty
- node_smaps_rollup_private

Check warning on line 98 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_private'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_private'?", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 98, "column": 6}}}, "severity": "INFO"}
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

Comment on lines +92 to +98
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
- type=anon
- type=file
- type=shmem
- node_smaps_rollup_shared
- type=clean
- type=dirty
- node_smaps_rollup_private
- node_smaps_rollup_private

- type=dirty
- type=clean
- node_smaps_rollup_referenced

Check warning on line 101 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_referenced'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_referenced'?", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 101, "column": 5}}}, "severity": "INFO"}
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_anonymous

Check warning on line 102 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_anonymous'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_anonymous'?", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 102, "column": 5}}}, "severity": "INFO"}
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_lazy_free

Check warning on line 103 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_lazy_free'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_lazy_free'?", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 103, "column": 5}}}, "severity": "INFO"}
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_anon_huge_pages

Check warning on line 104 in pipeline/inputs/mem-metrics.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_anon_huge_pages'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'node_smaps_rollup_anon_huge_pages'?", "location": {"path": "pipeline/inputs/mem-metrics.md", "range": {"start": {"line": 104, "column": 5}}}, "severity": "INFO"}
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_shmem_pmd_mapped
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_file_pmd_mapped
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_shared_hugetlb
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_private_hugetlb
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_swap
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_swap_pss
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

- node_smaps_rollup_locked
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint] reported by reviewdog 🐶
MD007/ul-indent Unordered list indentation [Expected: 0; Actual: 2]

Comment on lines +101 to +111
Copy link

@github-actions github-actions bot Nov 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[markdownlint-fix] reported by reviewdog 🐶

Suggested change
- node_smaps_rollup_referenced
- node_smaps_rollup_anonymous
- node_smaps_rollup_lazy_free
- node_smaps_rollup_anon_huge_pages
- node_smaps_rollup_shmem_pmd_mapped
- node_smaps_rollup_file_pmd_mapped
- node_smaps_rollup_shared_hugetlb
- node_smaps_rollup_private_hugetlb
- node_smaps_rollup_swap
- node_smaps_rollup_swap_pss
- node_smaps_rollup_locked
- node_smaps_rollup_referenced
- node_smaps_rollup_anonymous
- node_smaps_rollup_lazy_free
- node_smaps_rollup_anon_huge_pages
- node_smaps_rollup_shmem_pmd_mapped
- node_smaps_rollup_file_pmd_mapped
- node_smaps_rollup_shared_hugetlb
- node_smaps_rollup_private_hugetlb
- node_smaps_rollup_swap
- node_smaps_rollup_swap_pss
- node_smaps_rollup_locked

Loading