docs: document mTLS configuration for NiFi 2.x monitoring

docs/modules/nifi/pages/usage_guide/monitoring.adoc

@@ -2,19 +2,20 @@
 :description: The Stackable Operator for Apache NiFi automatically configures NiFi to export Prometheus metrics.
 :k8s-job: https://kubernetes.io/docs/concepts/workloads/controllers/job/
 :k8s-network-policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
+:prometheus-operator: https://prometheus-operator.dev/
 
-In November 2024, Apache NiFi released a new major version https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0[`2.0.0`].
+In November 2024, Apache NiFi released a new major version https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0[`2.0.0`,window=_blank].
 
 The NiFi `2.0.0` release changed the way of exposing Prometheus metrics significantly.
 The following steps explain on how to expose Metrics in NiFi versions `1.x.x` and `2.x.x`.
 
 == Configure metrics in NiFi `1.x.x`
 
 For NiFi versions `1.x.x`, the operator automatically configures NiFi to export Prometheus metrics.
-This is done by creating a {k8s-job}[Job] that connects to NiFi and configures a https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.26.0/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html[Prometheus Reporting Task].
+This is done by creating a {k8s-job}[Job,window=_blank] that connects to NiFi and configures a https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.26.0/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.html[Prometheus Reporting Task,window=_blank].
 
 IMPORTANT: Network access from the Job to NiFi is required.
-If you are running a Kubernetes with restrictive {k8s-network-policies}[NetworkPolicies], make sure to allow access from the Job to NiFi.
+If you are running a Kubernetes with restrictive {k8s-network-policies}[NetworkPolicies,window=_blank], make sure to allow access from the Job to NiFi.
 
 See xref:operators:monitoring.adoc[] for more details.
 
@@ -34,94 +35,83 @@ spec:
 
 == Configure metrics in NiFi `2.x.x`
 
-The Prometheus Reporting Task was removed in NiFi `2.x.x` in https://issues.apache.org/jira/browse/NIFI-13507[NIFI-13507].
-Metrics are now always exposed and can be scraped using the NiFi Pod FQDN and the HTTP path `/nifi-api/flow/metrics/prometheus`.
+The Prometheus Reporting Task was removed in NiFi `2.x.x` in https://issues.apache.org/jira/browse/NIFI-13507[NIFI-13507,window=_blank].
+Metrics are now always exposed and can be scraped using the NiFi `-metrics` Service and the HTTP path `/nifi-api/flow/metrics/prometheus`.
 
-For a deployed single node NiFi cluster called `simple-nifi`, containing a rolegroup called `default`, the metrics endpoint is reachable under:
+For a deployed NiFi cluster called `simple-nifi`, containing a rolegroup called `default`, the metrics endpoint is reachable under:
 
 ```
-https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
+https://simple-nifi-node-default-metrics.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus
 ```
 
-IMPORTANT: If NiFi is configured to do any user authentication, requests to the metric endpoint must be authenticated and authorized.
+NOTE: The above URL connects to one of the Pods, reachable through the specified Service, therefore scraping metrics produced by that Pod only.
+To scrape metrics from a particular Pod the FQDN of the Pod and the `-headless` Service need to be used. For example: `\https://simple-nifi-node-default-0.simple-nifi-node-default-headless.<namespace>.svc.cluster.local:8443/nifi-api/flow/metrics/prometheus`
 
-=== Authentication with NiFi `2.x.x`
-
-[IMPORTANT]
-===
-The NiFi metrics endpoints are behind a strong authentication mechanism which require credentials for each individual pod.
-===
+IMPORTANT: If NiFi is configured to do any user authentication, requests to the metrics endpoint must be authenticated and authorized.
 
-To authenticate, you can use a bearer token created by your NiFi instance e.g.
+=== Authentication with NiFi `2.x.x`
 
-[source,bash]
-----
-curl -X POST https://simple-nifi-node-default-0.simple-nifi-node-default.<namespace>.svc.cluster.local:8443/nifi-api/access/token -d 'username=<user>&password=<password>' -k
-----
+To authenticate against the NiFi `2.x.x` API, you can configure mTLS between NiFi and the client calling NiFi. For more information about authentication between
+Kubernetes Pods, check out the xref:home:secret-operator:index.adoc[Secret Operator documentation].
 
-where `-k` equals `verify=false` to allow self-signed certificates. The reply is your bearer token.
+For the following example, showing how to configure a Prometheus scraper for NiFi, we use the afore-mentioned method of configuring mTLS
+and utilizing the internally available `tls` xref:home:secret-operator:secretclass.adoc[SecretClass].
 
-The following example shows how to configure the Prometheus scraper to use the bearer token to authenticate against a NiFi pod.
+To generate a client certificate signed by the `tls` SecretClass CA trusted in NiFi, add the following `volume` and `volumeMount`
+to the Prometheus Pod.
 
-[source,yaml]
-----
----
-authorization: <1>
-  type: Bearer
-  credentials: "<Bearer Token>" <2>
-tls_config:
-  insecure_skip_verify: true
-static_configs:
-  - targets:
-    - '<pod>.<statefulset>.svc.cluster.local:8443' <3>
-metrics_path: '/nifi-api/flow/metrics/prometheus'
-scheme: https
-----
-<1> Use the `authorization` property instead if the `basic_auth`.
-<2> Add the previously obtained token here.
-<3> Static targets only scrapes one pod.
+IMPORTANT: If the {prometheus-operator}[Prometheus Operator,window=_blank] is being used to deploy Prometheus, there is currently a known bug, which prevents adding an additional Volume containing annotations on the volumeClaimTemplate. The bug is tracked in the https://github.com/prometheus-operator/prometheus-operator/issues/7709[Prometheus Operator Issues,window=_blank]. The annotations are necessary to configure the behavior of the Secret Operator. As a current workaround, until the issue is resolved, one could deploy an additional Pod only responsible for creating a TLS certificate as a Secret, which then can be used by the ServiceMonitor. This workaround is illustrated in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[`monitoring` Stack,window=_blank].
 
-or use it in a NiFi secret which should look like
 [source,yaml]
 ----
 ---
-apiVersion: v1
-kind: Secret
-metadata:
-  name: nifi-authorization-secret
-type: Opaque
-stringData:
-  nifi_token: "<Bearer_token>"
+prometheus: <1>
+  prometheusSpec:
+    volumes:
+      - name: tls
+        ephemeral:
+          volumeClaimTemplate:
+            metadata:
+              annotations:
+                secrets.stackable.tech/class: tls # <2>
+                secrets.stackable.tech/scope: pod,service=prometheus-kube-prometheus-prometheus # <3>
+            spec:
+              storageClassName: secrets.stackable.tech
+              accessModes:
+                - ReadWriteOnce
+              resources:
+                requests:
+                  storage: "1"
+    volumeMounts:
+      - name: tls
+        mountPath: /stackable/tls # <4>
 ----
+<1> This given configuration is set in the {prometheus-operator}docs/api-reference/api/#monitoring.coreos.com/v1.Prometheus[Prometheus resource,window=_blank] for the {prometheus-operator}[Prometheus Operator,window=_blank]
+<2> We use the `tls` SecretClass here, shipped with the Secret Operator, storing its CA in a Kubernetes Secret. Any other SecretClass can be used as well
+<3> We add the `service=prometheus-kube-prometheus-prometheus` scope for the Prometheus Service to be included in the `subjectAlternateName` of the TLS certificate. This particular Service name, used here, refers to the Prometheus Service deployed by the {prometheus-operator}[Prometheus Operator,window=_blank]
+<4> The path where the mTLS certificates are mounted to inside the Prometheus Pod
 
 If you want to use a `ServiceMonitor` you'd need to configure it as follows:
 
-// TODO: The ServiceMonitor should be switched to the -metrics service
-
 [source,yaml]
 ----
 ---
 apiVersion: monitoring.coreos.com/v1
 kind: ServiceMonitor
 metadata:
-  name: scrape-nifi2
+  name: scrape-nifi
   labels:
     stackable.tech/vendor: Stackable
     release: prometheus
 spec:
   endpoints:
-  - port: https
-    path: 'nifi-api/flow/metrics/prometheus'
+  - path: /nifi-api/flow/metrics/prometheus
+    port: https
     scheme: https
-    interval: 5s
-    tlsConfig:
-      insecureSkipVerify: true
-    authorization:
-      credentials: <1>
-        key: "nifi_token"
-        name: "nifi-authorization-secret"
-        optional: false
-      type: "Bearer"
+    tlsConfig: <1>
+      caFile: /stackable/tls/ca.crt
+      certFile: /stackable/tls/tls.crt
+      keyFile: /stackable/tls/tls.key
     relabelings: <2>
       - sourceLabels:
           - __meta_kubernetes_pod_name
@@ -130,15 +120,20 @@ spec:
           - __meta_kubernetes_pod_container_port_number
         targetLabel: __address__
         replacement: ${1}.${2}.${3}.svc.cluster.local:${4}
-        regex: (.+);(.+?)(?:-headless)?;(.+);(.+)
+        regex: (.+);(.+?)(?:-headless)?;(.+);(.+) # <3>
   selector:
     matchLabels:
       prometheus.io/scrape: "true"
   namespaceSelector:
     any: true
   jobLabel: app.kubernetes.io/instance
 ----
-<1> Authorization via Bearer Token stored in a secret
-<2> Relabel \\__address__ to be a FQDN rather then the IP-Address of target pod
+<1> In the TLS configuration of the ServiceMonitor, specify the paths to the mTLS files mounted to the Prometheus Pod
+<2> Relabel \\__address__ to be a FQDN rather then the IP-Address of the target Pod. This is currently necessary for scraping NiFi since it requires a DNS name to address the NiFi REST API
+<3> Currently, the NiFi StatefulSet only offers using FQDNs for NiFi Pods through the `-headless` Service, which is why we use the `-headless` Service instead of the `-metrics` Service for scraping NiFi metrics
 
 NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-headless` thus we need to regex this suffix.
+
+The described example is part of the https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus.yaml[Prometheus,window=_blank]
+and https://github.com/stackabletech/demos/blob/main/stacks/monitoring/prometheus-service-monitors.yaml[ServiceMonitor,window=_blank] manifests
+being used in the https://github.com/stackabletech/demos/blob/main/stacks/monitoring[`monitoring` Stack,window=_blank] of the https://github.com/stackabletech/demos[`demos` repository,window=_blank].