-
Notifications
You must be signed in to change notification settings - Fork 6
Monitoring changes in Apache NiFi 2.x
Apache NiFi 2.x has changed the monitoring process. In Apache NiFi 1.x, PrometheusReportingTask was used to get metrics from NiFi. In 2.x, such a Reporting Task is not available. Instead, REST API Metrics is used.
REST API Metrics is available at:
GET /nifi-api/flow/metrics/prometheus
| Query parameter | Required | Description |
|---|---|---|
| includedRegistries | N | Set of included metrics registries. Duplicate the parameter to include multiple registries. All registries are included by default. Available values: "NIFI", "JVM", "BULLETIN", "CONNECTION", "CLUSTER". |
| sampleName | N | Regular Expression Pattern to be applied against the sample name field. |
| sampleLabelValue | N | Regular Expression Pattern to be applied against the sample label value field. |
To receive metrics, the user must have permission to view the UI:
resource="/flow" action="R"
Changes:
- Instance label
- Control of monitoring data size
- HTTP support
In the case of PrometheusReportingTask, it was possible to set the value using expression language for the instance label using the Instance ID property. Also, the Instance ID property had a default value of ${hostname(true)}. In the case of the Metrics REST API, this is not possible. The instance label value in the REST API is the NiFi instance id and is generated automatically.
In PrometheusReportingTask, the Metrics Reporting Strategy property allowed you to control the number of exposed metrics. The Metrics REST API does not allow you to limit the number of returned metrics. In PrometheusReportingTask, there were on average 20 metrics per component. In the case of the Metrics REST API, there are 42 metrics per component.
For example, if a NiFi instance contains 10,000 components in version 1.x, the average number of metrics will be 200,000. In case of 2.x, the number of metrics will be 420,000.
One way to solve the problem of the number of metrics in version 2.x is to create a custom reporting task with support for the level of detail of the indicators.
Such an average number of metrics can overload the monitoring system. The includedRegistries parameter allows you to control which registries the metrics will be retrieved from. For example, if 'includedRegistries=JVM', then only metrics about the JVM will be retrieved.
| Parameter | NiFi 1.x | NiFi 2.x |
|---|---|---|
| Port |
Prometheus Metrics Endpoint Port property |
HTTP/HTTPS Apache NiFi port |
| Protocol | HTTP/HTTPS depending on reporting task configuration | HTTP/HTTPS depending on Apache NiFi configuration |
| URL path | /metrics |
/nifi-api/flow/metrics/prometheus |
| TLS CA certificate | client.p12 | client.p12 |
| TLS client certificate | nifi-cert.pem | nifi-cert.pem |
For Prometheus to work with Apache NiFi 1.x the HTTPS protocol was optional. In the case of Apache NiFi 2.x, it is mandatory, if Apache NiFi instance is secured, which will be true for all production-level systems.