Monitoring changes in Apache NiFi 2.x

Monitoring changes in Apache NiFi 2.x

Apache NiFi 2.x has changed the monitoring process. In Apache NiFi 1.x, PrometheusReportingTask was used to get metrics from NiFi. In 2.x, such a Reporting Task is not available. Instead, REST API Metrics is used.

Metrics API

REST API Metrics is available at: GET /nifi-api/flow/metrics/prometheus

Query parameter	Required	Description
includedRegistries	N	Set of included metrics registries. Duplicate the parameter to include multiple registries. All registries are included by default. Available values: "NIFI", "JVM", "BULLETIN", "CONNECTION", "CLUSTER".
sampleName	N	Regular Expression Pattern to be applied against the sample name field.
sampleLabelValue	N	Regular Expression Pattern to be applied against the sample label value field.

To receive metrics, the user must have permission to view the UI: resource="/flow" action="R"

Breaking changes

Changes:

1. Instance label
2. Control of monitoring data size
3. HTTP support

Instance label

In the case of PrometheusReportingTask, it was possible to set the value using expression language for the instance label using the Instance ID property. Also, the Instance ID property had a default value of ${hostname(true)}. In the case of the Metrics REST API, this is not possible. The instance label value in the REST API is the NiFi instance id and is generated automatically.

Control of monitoring data size

In PrometheusReportingTask, the Metrics Reporting Strategy property allowed you to control the number of exposed metrics. The Metrics REST API does not allow you to limit the number of returned metrics. In PrometheusReportingTask, there were on average 20 metrics per component. In the case of the Metrics REST API, there are 42 metrics per component.

For example, if a NiFi instance contains 10,000 components in version 1.x, the average number of metrics will be 200,000. In case of 2.x, the number of metrics will be 420,000.

One way to solve the problem of the number of metrics in version 2.x is to create a custom reporting task with support for the level of detail of the indicators.

Such an average number of metrics can overload the monitoring system. The includedRegistries parameter allows you to control which registries the metrics will be retrieved from. For example, if 'includedRegistries=JVM', then only metrics about the JVM will be retrieved.

HTTP support

Parameter	NiFi 1.x	NiFi 2.x
Port	Prometheus Metrics Endpoint Port property	HTTP/HTTPS Apache NiFi port
Protocol	HTTP/HTTPS depending on reporting task configuration	HTTP/HTTPS depending on Apache NiFi configuration
URL path	/metrics	/nifi-api/flow/metrics/prometheus
TLS CA certificate	client.p12	client.p12
TLS client certificate	nifi-cert.pem	nifi-cert.pem

For Prometheus to work with Apache NiFi 1.x the HTTPS protocol was optional. In the case of Apache NiFi 2.x, it is mandatory, if Apache NiFi instance is secured, which will be true for all production-level systems.