Skip to content

Commit 29b6ec5

Browse files
xeniapemaltesander
xeniape
and
maltesander
authored
docs: document mTLS configuration for NiFi 2.x monitoring (#832)
* docs: document mTLS configuration for NiFi 2.x monitoring * update ServiceMonitor manifest * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> * Update docs/modules/nifi/pages/usage_guide/monitoring.adoc Co-authored-by: Malte Sander <[email protected]> --------- Co-authored-by: Malte Sander <[email protected]>
1 parent f2380a1 commit 29b6ec5

File tree

1 file changed

+62
-66
lines changed

1 file changed

+62
-66
lines changed

‎docs/modules/nifi/pages/usage_guide/monitoring.adoc

Lines changed: 62 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,20 @@
22
:description: The Stackable Operator for Apache NiFi automatically configures NiFi to export Prometheus metrics.
33
:k8s-job: https://kubernetes.io/docs/concepts/workloads/controllers/job/
44
:k8s-network-policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
5+
:prometheus-operator: https://prometheus-operator.dev/
56

6-
In November 2024, Apache NiFi released a new major version https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0[`2.0.0`].
7+
In November 2024, Apache NiFi released a new major version https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0[`2.0.0`,window=_blank].
78
89
The NiFi `2.0.0` release changed the way of exposing Prometheus metrics significantly.
910
The following steps explain on how to expose Metrics in NiFi versions `1.x.x` and `2.x.x`.
1011
1112
== Configure metrics in NiFi `1.x.x`
1213
1314
For NiFi versions `1.x.x`, the operator automatically configures NiFi to export Prometheus metrics.
14-
This is done by creating a {k8s-job}[Job] that connects to NiFi and configures a https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.26.0/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html[Prometheus Reporting Task].
15+
This is done by creating a {k8s-job}[Job,window=_blank] that connects to NiFi and configures a https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.26.0/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html[Prometheus Reporting Task,window=_blank].
1516
1617
IMPORTANT: Network access from the Job to NiFi is required.
17-
If you are running a Kubernetes with restrictive {k8s-network-policies}[NetworkPolicies], make sure to allow access from the Job to NiFi.
18+
If you are running a Kubernetes with restrictive {k8s-network-policies}[NetworkPolicies,window=_blank], make sure to allow access from the Job to NiFi.
1819
1920
See xref:operators:monitoring.adoc[] for more details.
2021
@@ -34,111 +35,106 @@ spec:
3435
3536
== Configure metrics in NiFi `2.x.x`
3637
37-
The Prometheus Reporting Task was removed in NiFi `2.x.x` in https://issues.apache.org/jira/browse/NIFI-13507[NIFI-13507].
38-
Metrics are now always exposed and can be scraped using the NiFi Pod FQDN and the HTTP path `/nifi-api/flow/metrics/prometheus`.
38+
The Prometheus Reporting Task was removed in NiFi `2.x.x` in https://issues.apache.org/jira/browse/NIFI-13507[NIFI-13507,window=_blank].
39+
Metrics are now always exposed and can be scraped using the NiFi `metrics` Service and the HTTP path `/nifi-api/flow/metrics/prometheus`.
3940
40-
For a deployed single node NiFi cluster called `simple-nifi`, containing a rolegroup called `default`, the metrics endpoint is reachable under:
41+
For a deployed NiFi cluster called `simple-nifi`, containing a rolegroup called `default`, the metrics endpoint is reachable under:
4142
4243
```
43-
https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
44+
https://simple-nifi-node-default-metrics.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
4445
```
4546
46-
IMPORTANT: If NiFi is configured to do any user authentication, requests to the metric endpoint must be authenticated and authorized.
47+
NOTE: The above URL connects to one of the Pods, reachable through the specified Service, therefore scraping metrics produced by that Pod only.
48+
To scrape metrics from a particular Pod, the FQDN of the Pod and the `headless` Service need to be used. For example: `\https://simple-nifi-node-default-0.simple-nifi-node-default-headless.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus`
4749
48-
=== Authentication with NiFi `2.x.x`
49-
50-
[IMPORTANT]
51-
===
52-
The NiFi metrics endpoints are behind a strong authentication mechanism which require credentials for each individual pod.
53-
===
50+
IMPORTANT: If NiFi is configured to do any user authentication, requests to the metrics endpoint must be authenticated and authorized.
5451
55-
To authenticate, you can use a bearer token created by your NiFi instance e.g.
52+
=== Authentication with NiFi `2.x.x`
5653
57-
[source,bash]
58-
----
59-
curl -X POST https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/access/token -d 'username=<user>&password=<password>' -k
60-
----
54+
To authenticate against the NiFi `2.x.x` API, you can configure mTLS between NiFi and the client calling NiFi. For more information about authentication between
55+
Kubernetes Pods, check out the xref:home:secret-operator:index.adoc[Secret Operator documentation].
6156
62-
where `-k` equals `verify=false` to allow self-signed certificates. The reply is your bearer token.
57+
The following example illustrates the configuration of a Prometheus scraper for NiFi, using the aforementioned method of configuring mTLS
58+
and utilizing the internally available `tls` xref:home:secret-operator:secretclass.adoc[SecretClass].
6359
64-
The following example shows how to configure the Prometheus scraper to use the bearer token to authenticate against a NiFi pod.
60+
To generate a client certificate signed by the `tls` SecretClass CA trusted in NiFi, add the following `volume` and `volumeMount`
61+
to the Prometheus Pod.
6562
66-
[source,yaml]
67-
----
68-
---
69-
authorization: <1>
70-
type: Bearer
71-
credentials: "<Bearer Token>" <2>
72-
tls_config:
73-
insecure_skip_verify: true
74-
static_configs:
75-
- targets:
76-
- '<pod>.<statefulset>.svc.cluster.local:8443' <3>
77-
metrics_path: '/nifi-api/flow/metrics/prometheus'
78-
scheme: https
79-
----
80-
<1> Use the `authorization` property instead if the `basic_auth`.
81-
<2> Add the previously obtained token here.
82-
<3> Static targets only scrapes one pod.
63+
IMPORTANT: If the {prometheus-operator}[Prometheus Operator,window=_blank] is used to deploy Prometheus, there is currently a known bug, which prevents adding an additional Volume containing annotations on the volumeClaimTemplate. The bug is tracked in the https://github.com/prometheus-operator/prometheus-operator/issues/7709[Prometheus Operator Issues,window=_blank]. The annotations are necessary to configure the behavior of the Secret Operator. As a current workaround, until the issue is resolved, one could deploy an additional Pod only responsible for creating a TLS certificate as a Secret, which then can be used by the ServiceMonitor. This workaround is illustrated in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[`monitoring` Stack,window=_blank].
8364
84-
or use it in a NiFi secret which should look like
8565
[source,yaml]
8666
----
8767
---
88-
apiVersion: v1
89-
kind: Secret
90-
metadata:
91-
name: nifi-authorization-secret
92-
type: Opaque
93-
stringData:
94-
nifi_token: "<Bearer_token>"
68+
prometheus: <1>
69+
prometheusSpec:
70+
volumes:
71+
- name: tls
72+
ephemeral:
73+
volumeClaimTemplate:
74+
metadata:
75+
annotations:
76+
secrets.stackable.tech/class: tls # <2>
77+
secrets.stackable.tech/scope: pod,service=prometheus-kube-prometheus-prometheus # <3>
78+
spec:
79+
storageClassName: secrets.stackable.tech
80+
accessModes:
81+
- ReadWriteOnce
82+
resources:
83+
requests:
84+
storage: "1"
85+
volumeMounts:
86+
- name: tls
87+
mountPath: /stackable/tls # <4>
9588
----
89+
<1> This given configuration is set in the {prometheus-operator}docs/api-reference/api/#monitoring.coreos.com/v1.Prometheus[Prometheus resource,window=_blank] for the {prometheus-operator}[Prometheus Operator,window=_blank]
90+
<2> The `tls` SecretClass created by the Secret Operator, storing its CA in a Kubernetes Secret. Any other SecretClass can be used as well
91+
<3> The `service=prometheus-kube-prometheus-prometheus` scope is added to include the `subjectAlternateName` of the Prometheus Service in the generated TLS certificate. This particular Service name, used here, refers to the Prometheus Service deployed by the {prometheus-operator}[Prometheus Operator,window=_blank]
92+
<4> The path where the mTLS certificates are mounted inside the Prometheus Pod
9693
9794
If you want to use a `ServiceMonitor` you'd need to configure it as follows:
9895
99-
// TODO: The ServiceMonitor should be switched to the -metrics service
100-
10196
[source,yaml]
10297
----
10398
---
10499
apiVersion: monitoring.coreos.com/v1
105100
kind: ServiceMonitor
106101
metadata:
107-
name: scrape-nifi2
102+
name: scrape-nifi-2
108103
labels:
109104
stackable.tech/vendor: Stackable
110105
release: prometheus
111106
spec:
112107
endpoints:
113-
- port: https
114-
path: 'nifi-api/flow/metrics/prometheus'
108+
- path: /nifi-api/flow/metrics/prometheus
109+
port: https
115110
scheme: https
116-
interval: 5s
117-
tlsConfig:
118-
insecureSkipVerify: true
119-
authorization:
120-
credentials: <1>
121-
key: "nifi_token"
122-
name: "nifi-authorization-secret"
123-
optional: false
124-
type: "Bearer"
125-
relabelings: <2>
111+
tlsConfig: # <1>
112+
caFile: /stackable/tls/ca.crt
113+
certFile: /stackable/tls/tls.crt
114+
keyFile: /stackable/tls/tls.key
115+
relabelings: # <2>
126116
- sourceLabels:
127117
- __meta_kubernetes_pod_name
128118
- __meta_kubernetes_service_name
129119
- __meta_kubernetes_namespace
130120
- __meta_kubernetes_pod_container_port_number
131121
targetLabel: __address__
132-
replacement: ${1}.${2}.${3}.svc.cluster.local:${4}
133-
regex: (.+);(.+?)(?:-headless)?;(.+);(.+)
122+
replacement: ${1}.${2}-headless.${3}.svc.cluster.local:${4} # <3>
123+
regex: (.+);(.+?)(?:-metrics)?;(.+);(.+)
134124
selector:
135125
matchLabels:
126+
stackable.tech/vendor: Stackable
136127
prometheus.io/scrape: "true"
137128
namespaceSelector:
138129
any: true
139130
jobLabel: app.kubernetes.io/instance
140131
----
141-
<1> Authorization via Bearer Token stored in a secret
142-
<2> Relabel \\__address__ to be a FQDN rather then the IP-Address of target pod
132+
<1> In the TLS configuration of the ServiceMonitor, specify the location of the cert and key files mounted into the Prometheus Pod
133+
<2> Relabel `__address__` to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary to scrape NiFi, since it requires a DNS name to address the NiFi REST API
134+
<3> Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the `headless` Service, which is why we use the `headless` Service instead of the `metrics` Service to scrape NiFi metrics
135+
136+
NOTE: The SDP exposes a dedicated `metrics` Service since the xref:listener-operator:listener.adoc[Listener integration].
143137

144-
NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-headless` thus we need to regex this suffix.
138+
The described example is part of the https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus.yaml[Prometheus,window=_blank]
139+
and https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus-service-monitors.yaml[ServiceMonitor,window=_blank] manifests
140+
used in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[monitoring stack,window=_blank] of the https://github.com/stackabletech/demos[demos repository,window=_blank].

0 commit comments

Comments
 (0)