Unify the 6 different Prometheus metrics provided by flowlogs-pipeline or add at least a global prefix for all metrics

[Nachtfalkeaw]

Hello,

I collect prometheus metrics from this endpoint and I receive 6 different prefixes of metrics. I would expect a possibility to prefix all metrics of flowlogs-pipline with the same prefix.

metricsSettings:
  suppressGoMetrics: false
  prefix: flp_operational_
  port: 9102

Result examples:

# HELP flow_decoder_count Decoder processed count.
# TYPE flow_decoder_count counter
flow_decoder_count{name="NetFlow",worker="0"} 6


# HELP flp_operational_ingest_batch_size_bytes Ingested batch size distribution, in bytes
# TYPE flp_operational_ingest_batch_size_bytes summary
flp_operational_ingest_batch_size_bytes{stage="ipfix_netflow_ingest",quantile="0.5"} NaN
flp_operational_ingest_batch_size_bytes{stage="ipfix_netflow_ingest",quantile="0.95"} NaN
flp_operational_ingest_batch_size_bytes_sum{stage="ipfix_netflow_ingest"} 0
flp_operational_ingest_batch_size_bytes_count{stage="ipfix_netflow_ingest"} 0

# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.805e-05
go_gc_duration_seconds{quantile="0.25"} 6.7171e-05
go_gc_duration_seconds{quantile="0.5"} 0.000186544
go_gc_duration_seconds{quantile="0.75"} 0.000226612
go_gc_duration_seconds{quantile="1"} 0.000251489
go_gc_duration_seconds_sum 0.000870278
go_gc_duration_seconds_count 6

# HELP net_conntrack_dialer_conn_closed_total Total number of connections closed which originated from the dialer of a given name.
# TYPE net_conntrack_dialer_conn_closed_total counter
net_conntrack_dialer_conn_closed_total{dialer_name="promtail"} 5

# HELP netobserv_loki_dropped_bytes_total Number of bytes dropped because failed to be sent to the ingester after all retries.
# TYPE netobserv_loki_dropped_bytes_total counter
netobserv_loki_dropped_bytes_total{host="prometheus.inet.bundesbank.de:3100"} 0

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.1

6 different metrics prefixes. I would expect to only have one 8except for the go specific metrics).

they could all start with
flp_

an then have different variations and the second part if this is really needed. But maybe useful to differentiate between different fetures.
flp_flow_
flp_operational_
flp_net_conntrack_
flp_netobserv_

As an example Prometheus "node_exporter" starts with "node_" and all metrics of this exporter are behind "node_*"

[jotak]

Hi,

We need to make a distinction here, between metrics provided by FLP and metrics provided by dependent libraries (in particular by the prometheus go client). I think your suggestion is fine for the metrics provided by FLP directly. Among the ones that you mention, it's only:

• flp_operational_*
• netobserv_loki_dropped_bytes_total
  I agree for these ones to see if we can make them more consistent - with the "flp_" prefix, generating "flp_operational_*" and "flp_loki_dropped_bytes_total".

But for the other ones, they belong to other components (flow_decoder_count I believe is provided by goflow2 which is used here under the cover, and the other ones are provided by the prometheus go client). Having these metrics not prefixed with FLP is done on purpose, to let users run queries that work across multiple programs. For instance, it lets you compare the CPU time spent across all eligible programs. Typically, different labels would be applied for each program that expose these metrics (such as "job" label, or "instance", etc. - that depends on your prometheus scraping configuration).

[Nachtfalkeaw]

I think net_conntrack_ is flp, too, isn't it?
I thought it is related to bidirectional Connection, heartbeat, new Connections etc.

Maybe it is possible to add a Label to all These metrics except the Go Client metrics to make clear it is related or relevant for flp application.

But I understand what you explained and makes Sense to me.

[jotak]

I just checked the conntrack one, it comes from the prometheus client which itself pulls this: https://github.com/mwitkow/go-conntrack/blob/2f068394615f73e460c2f3d2c158b0ad9321cadb/dialer_reporter.go#L49
It can be confusing with FLP's own conversation tracking system, but it's actually different