plugin: in_proc_metrics

[pwhelan]

Add a new plugin which logs the metrics for a single process.

---

Testing

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

• Documentation required for this feature

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[pwhelan]

Here is a valgrind log where I opened and closed some instances of htop.

valgrind.log

[pwhelan]

Here is a simple configuration file that shows statistics from all the htop processes that are running:

[INPUT]
    Name    proc_metrics
    Tag        metrics
    Process htop

[OUTPUT]
    Name stdout
    Match metrics

[pwhelan]

Here is a log of fluent-bit running with verbose logging:

fluent-bit.log

[pwhelan]

Here is the PR for the documentation: fluent/fluent-bit-docs#609.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[edsiper]

thanks for this PR.

I am thinking about what kind of process metrics exporter exists already in the Prometheus world, basically, I would like to make sure that our metrics names and labels can match what other projects are doing to avoid the end-user re-configure their dashboards. I found some examples:

• https://github.com/ncabatoff/process-exporter
• https://github.com/tgulacsi/prometheus_scollector

not sure if there is other options, which one we should follow ?

[edsiper]

@patrick-stephens are you familiar with any of the projects mentioned above ?

[patrick-stephens]

The https://github.com/ncabatoff/process-exporter is the one I've generally seen people use and briefly experimented with but am not an expert in I'm afraid.

I'd also watch out for container/kubernetes issues with trying to mount /proc as well if that's the approach you're taking. How is this intended to function in a container world? Are we trying to retrieve metrics about processes on the host, pod or just inside the container here?

The docs I think need to cover what you need to do to get it working in Kubernetes and a standalone container. Particularly with security constraints, how do we enable the capability to do it or at least flag up it may not work without extra effort.

We probably need to follow the approach Kubernetes is taking there as well: https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/
I'm not sure how stable this is but there are various guides on what the kubelet offers for metrics as well as the other components.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[pwhelan]

this seems to have little to no chance of getting in and already seems to have drifted considerably. closing it for now.