-
Notifications
You must be signed in to change notification settings - Fork 540
pipeline: input: add prometheus textfile plugin doc #2036
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,112 @@ | ||||||||
| # Prometheus Textfile | ||||||||
|
Check warning on line 1 in pipeline/inputs/prometheus-textfile.md
|
||||||||
|
|
||||||||
| The **prometheus_textfile** input plugin allows Fluent Bit to read metrics from Prometheus text format files (.prom files) on the local filesystem. This plugin is particularly useful for collecting custom metrics that are written to files by external applications or scripts, similar to the Prometheus Node Exporter textfile collector. | ||||||||
|
Check warning on line 3 in pipeline/inputs/prometheus-textfile.md
|
||||||||
|
|
||||||||
| ## Configuration Parameters | ||||||||
|
Check warning on line 5 in pipeline/inputs/prometheus-textfile.md
|
||||||||
|
|
||||||||
| | Key | Description | Default | | ||||||||
| |-----|-------------|---------| | ||||||||
| | path | File or directory path pattern. Supports glob patterns with `*` wildcard (e.g., `/var/lib/prometheus/*.prom`) | | | ||||||||
|
Check failure on line 9 in pipeline/inputs/prometheus-textfile.md
|
||||||||
| | scrape_interval | Interval in seconds between file scans | 10s | | ||||||||
|
Check warning on line 10 in pipeline/inputs/prometheus-textfile.md
|
||||||||
|
|
||||||||
| ## Getting Started | ||||||||
|
Check warning on line 12 in pipeline/inputs/prometheus-textfile.md
|
||||||||
|
|
||||||||
| ### Simple Configuration | ||||||||
|
Check warning on line 14 in pipeline/inputs/prometheus-textfile.md
|
||||||||
|
|
||||||||
| The following configuration will monitor `/var/lib/prometheus/textfile` directory for `.prom` files every 15 seconds: | ||||||||
|
|
||||||||
| ```yaml | ||||||||
| pipeline: | ||||||||
| inputs: | ||||||||
| - name: prometheus_textfile | ||||||||
| tag: custom_metrics | ||||||||
| path: '/var/lib/prometheus/textfile/*.prom' | ||||||||
| scrape_interval: 15 | ||||||||
| outputs: | ||||||||
| - name: prometheus_exporter | ||||||||
| match: custom_metrics | ||||||||
| host: 192.168.100.61 | ||||||||
| port: 2021 | ||||||||
| ``` | ||||||||
|
|
||||||||
| ## Prometheus Text Format | ||||||||
|
|
||||||||
| The plugin expects files to be in the standard Prometheus text exposition format. Here's an example of a valid `.prom` file: | ||||||||
|
|
||||||||
| ``` | ||||||||
| # HELP custom_counter_total A custom counter metric | ||||||||
| # TYPE custom_counter_total counter | ||||||||
| custom_counter_total{instance="server1",job="myapp"} 42 | ||||||||
|
|
||||||||
| # HELP custom_gauge A custom gauge metric | ||||||||
| # TYPE custom_gauge gauge | ||||||||
| custom_gauge{environment="production"} 1.23 | ||||||||
|
|
||||||||
| # HELP custom_histogram_bucket A custom histogram | ||||||||
| # TYPE custom_histogram_bucket histogram | ||||||||
| custom_histogram_bucket{le="0.1"} 10 | ||||||||
| custom_histogram_bucket{le="0.5"} 25 | ||||||||
| custom_histogram_bucket{le="1.0"} 40 | ||||||||
| custom_histogram_bucket{le="+Inf"} 50 | ||||||||
| custom_histogram_sum 125.5 | ||||||||
| custom_histogram_count 50 | ||||||||
| ``` | ||||||||
|
|
||||||||
| ## Use Cases | ||||||||
|
|
||||||||
| ### Custom Application Metrics | ||||||||
|
|
||||||||
| Applications can write custom metrics to `.prom` files, and this plugin will collect them: | ||||||||
|
|
||||||||
| ```bash | ||||||||
| # Script writes metrics to file | ||||||||
| echo "# HELP app_requests_total Total HTTP requests" > /var/lib/prometheus/textfile/app.prom | ||||||||
| echo "# TYPE app_requests_total counter" >> /var/lib/prometheus/textfile/app.prom | ||||||||
| echo "app_requests_total{status=\"200\"} 1500" >> /var/lib/prometheus/textfile/app.prom | ||||||||
| echo "app_requests_total{status=\"404\"} 23" >> /var/lib/prometheus/textfile/app.prom | ||||||||
| ``` | ||||||||
|
|
||||||||
| ### Batch Job Metrics | ||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [markdownlint] reported by reviewdog 🐶 |
||||||||
| Cron jobs or batch processes can write completion metrics: | ||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [markdownlint-fix] reported by reviewdog 🐶
Suggested change
|
||||||||
|
|
||||||||
| ```bash | ||||||||
| #!/bin/bash | ||||||||
| # Backup script writes completion metrics | ||||||||
| BACKUP_START=$(date +%s) | ||||||||
| # ... perform backup ... | ||||||||
| BACKUP_END=$(date +%s) | ||||||||
| DURATION=$((BACKUP_END - BACKUP_START)) | ||||||||
|
|
||||||||
| cat > /var/lib/prometheus/textfile/backup.prom << EOF | ||||||||
| # HELP backup_duration_seconds Time taken to complete backup | ||||||||
| # TYPE backup_duration_seconds gauge | ||||||||
| backup_duration_seconds ${DURATION} | ||||||||
|
|
||||||||
| # HELP backup_last_success_timestamp_seconds Last successful backup timestamp | ||||||||
| # TYPE backup_last_success_timestamp_seconds gauge | ||||||||
| backup_last_success_timestamp_seconds ${BACKUP_END} | ||||||||
| EOF | ||||||||
| ``` | ||||||||
|
|
||||||||
| ### System Integration | ||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [markdownlint] reported by reviewdog 🐶 |
||||||||
| External monitoring tools can write metrics that Fluent Bit will collect and forward. | ||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [markdownlint-fix] reported by reviewdog 🐶
Suggested change
|
||||||||
|
|
||||||||
| ## Integration with Other Plugins | ||||||||
|
|
||||||||
| ### OpenTelemetry Destination | ||||||||
|
|
||||||||
| ```yaml | ||||||||
| pipeline: | ||||||||
| inputs: | ||||||||
| - name: prometheus_textfile | ||||||||
| tag: textfile_metrics | ||||||||
| path: /var/lib/prometheus/textfile | ||||||||
| - name: node_exporter_metrics | ||||||||
| tag: system_metrics | ||||||||
| scrape_interval: 15 | ||||||||
| outputs: | ||||||||
| - name: opentelemetry | ||||||||
| match: '*' | ||||||||
| host: 192.168.56.4 | ||||||||
| port: 2021 | ||||||||
| ``` | ||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[markdownlint] reported by reviewdog 🐶
MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"]