Add NiFi Agent Integration PoC (AI-6667)

nifi/poc/README.md

@@ -0,0 +1,58 @@
+# Apache NiFi Agent Integration — PoC
+
+Proof-of-concept demonstrating that the NiFi REST API provides all metrics
+needed for a Datadog Agent integration. Runs NiFi 2.8.0 in Docker and queries
+every monitoring endpoint the integration will use.
+
+**RFC**: [AI-6667](https://datadoghq.atlassian.net/browse/AI-6667)
+**Epic**: [AI-6662](https://datadoghq.atlassian.net/browse/AI-6662)
+
+## Quick Start
+
+```bash
+cd nifi/poc
+./scripts/run-poc.sh
+```
+
+This will:
+1. Start NiFi 2.8.0 in Docker (HTTPS, single-user auth)
+2. Wait for the API to become available (~60-90s)
+3. Create test flows via REST API:
+   - **Happy path**: GenerateFlowFile → LogMessage (produces throughput metrics)
+   - **Error path**: GenerateFlowFile → PutFile /nonexistent (produces ERROR bulletins)
+4. Wait for data to flow (30s)
+5. Query all monitoring endpoints and save responses to `responses/`
+
+## Endpoints Tested
+
+| Endpoint | Purpose | Integration Use |
+|----------|---------|-----------------|
+| `GET /flow/about` | Version detection | Cached tag, connectivity check |
+| `GET /system-diagnostics` | JVM heap, GC, threads, repos | System health metrics |
+| `GET /flow/status` | Running/stopped/invalid counts | Flow summary metrics |
+| `GET /flow/process-groups/root/status?recursive=true` | All component data in one call | Process group, processor, connection metrics |
+| `GET /flow/cluster/summary` | Cluster vs standalone detection | Cluster health metrics |
+| `GET /flow/bulletin-board` | Errors and warnings | Datadog events |
+| `GET /system-diagnostics/jmx-metrics` | JMX metrics via REST | Returns empty unless `nifi.web.jmx.metrics.allowed.filter.pattern` is configured; not used by integration |
+
+## Authentication
+
+NiFi 2.x requires HTTPS and authentication. This PoC uses single-user mode:
+
+```
+Username: admin
+Password: ctsBtRBKHRAx69EqUghvvgEvjnaLjFEB
+```
+
+Token obtained via `POST /access/token` (returns JWT, 12h expiry).
+
+## Teardown
+
+```bash
+cd nifi/poc
+docker compose down -v
+```
+
+## NiFi UI
+
+While running: https://localhost:8443/nifi/ (accept self-signed cert)

nifi/poc/docker-compose.yaml

@@ -0,0 +1,14 @@
+services:
+  nifi:
+    image: apache/nifi:2.8.0
+    ports:
+      - "8443:8443"
+    environment:
+      SINGLE_USER_CREDENTIALS_USERNAME: admin
+      SINGLE_USER_CREDENTIALS_PASSWORD: ctsBtRBKHRAx69EqUghvvgEvjnaLjFEB
+    healthcheck:
+      test: ["CMD-SHELL", "curl -sk -o /dev/null -w '%{http_code}' https://localhost:8443/nifi-api/flow/about | grep -qv 000"]
+      interval: 10s
+      timeout: 5s
+      retries: 30
+      start_period: 60s

nifi/poc/responses/about.json

@@ -0,0 +1,10 @@
+{
+  "about": {
+    "title": "NiFi",
+    "version": "2.8.0",
+    "uri": "https://localhost:8443/nifi-api/",
+    "contentViewerUrl": "https://localhost:8443/nifi/#/content-viewer",
+    "timezone": "UTC",
+    "buildTag": "rel/nifi-2.8.0"
+  }
+}

nifi/poc/responses/bulletin-board.json

@@ -0,0 +1,70 @@
+{
+  "bulletinBoard": {
+    "bulletins": [
+      {
+        "id": 0,
+        "groupId": "0c6f3d7b-019d-1000-5edf-dc0f4c54ff63",
+        "sourceId": "0c6f483e-019d-1000-0127-4e7850192d3e",
+        "timestamp": "18:08:33 UTC",
+        "timestampIso": "2026-03-20T18:08:33.065Z",
+        "canRead": true,
+        "bulletin": {
+          "id": 0,
+          "category": "Log Message",
+          "groupId": "0c6f3d7b-019d-1000-5edf-dc0f4c54ff63",
+          "sourceId": "0c6f483e-019d-1000-0127-4e7850192d3e",
+          "sourceName": "Fail Writer",
+          "level": "ERROR",
+          "message": "PutFile[id=0c6f483e-019d-1000-0127-4e7850192d3e] Penalizing FlowFile[filename=9c43fd5b-1fc5-4937-8759-48bad37b5364] and transferring to failure: java.nio.file.AccessDeniedException: /nonexistent",
+          "timestamp": "18:08:33 UTC",
+          "timestampIso": "2026-03-20T18:08:33.065Z",
+          "sourceType": "PROCESSOR",
+          "stackTrace": "java.nio.file.AccessDeniedException: /nonexistent\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:462)\n\tat java.base/java.nio.file.Files.createDirectory(Files.java:700)\n\tat java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:808)\n\tat java.base/java.nio.file.Files.createDirectories(Files.java:794)\n\tat org.apache.nifi.processors.standard.PutFile.onTrigger(PutFile.java:242)\n\tat org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)\n\tat org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1285)\n\tat org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)\n\tat org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)\n\tat org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"
+        }
+      },
+      {
+        "id": 1,
+        "groupId": "0c6f3d7b-019d-1000-5edf-dc0f4c54ff63",
+        "sourceId": "0c6f483e-019d-1000-0127-4e7850192d3e",
+        "timestamp": "18:08:43 UTC",
+        "timestampIso": "2026-03-20T18:08:43.031Z",
+        "canRead": true,
+        "bulletin": {
+          "id": 1,
+          "category": "Log Message",
+          "groupId": "0c6f3d7b-019d-1000-5edf-dc0f4c54ff63",
+          "sourceId": "0c6f483e-019d-1000-0127-4e7850192d3e",
+          "sourceName": "Fail Writer",
+          "level": "ERROR",
+          "message": "PutFile[id=0c6f483e-019d-1000-0127-4e7850192d3e] Penalizing FlowFile[filename=9ab0f226-d548-4b5a-b1ff-7b7b0802b7c1] and transferring to failure: java.nio.file.AccessDeniedException: /nonexistent",
+          "timestamp": "18:08:43 UTC",
+          "timestampIso": "2026-03-20T18:08:43.031Z",
+          "sourceType": "PROCESSOR",
+          "stackTrace": "java.nio.file.AccessDeniedException: /nonexistent\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:462)\n\tat java.base/java.nio.file.Files.createDirectory(Files.java:700)\n\tat java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:808)\n\tat java.base/java.nio.file.Files.createDirectories(Files.java:794)\n\tat org.apache.nifi.processors.standard.PutFile.onTrigger(PutFile.java:242)\n\tat org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)\n\tat org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1285)\n\tat org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)\n\tat org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)\n\tat org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"
+        }
+      },
+      {
+        "id": 2,
+        "groupId": "0c6f3d7b-019d-1000-5edf-dc0f4c54ff63",
+        "sourceId": "0c6f483e-019d-1000-0127-4e7850192d3e",
+        "timestamp": "18:08:53 UTC",
+        "timestampIso": "2026-03-20T18:08:53.032Z",
+        "canRead": true,
+        "bulletin": {
+          "id": 2,
+          "category": "Log Message",
+          "groupId": "0c6f3d7b-019d-1000-5edf-dc0f4c54ff63",
+          "sourceId": "0c6f483e-019d-1000-0127-4e7850192d3e",
+          "sourceName": "Fail Writer",
+          "level": "ERROR",
+          "message": "PutFile[id=0c6f483e-019d-1000-0127-4e7850192d3e] Penalizing FlowFile[filename=f9d78ec8-f4b7-460c-a9bd-51ca20041dfd] and transferring to failure: java.nio.file.AccessDeniedException: /nonexistent",
+          "timestamp": "18:08:53 UTC",
+          "timestampIso": "2026-03-20T18:08:53.032Z",
+          "sourceType": "PROCESSOR",
+          "stackTrace": "java.nio.file.AccessDeniedException: /nonexistent\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:462)\n\tat java.base/java.nio.file.Files.createDirectory(Files.java:700)\n\tat java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:808)\n\tat java.base/java.nio.file.Files.createDirectories(Files.java:794)\n\tat org.apache.nifi.processors.standard.PutFile.onTrigger(PutFile.java:242)\n\tat org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)\n\tat org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1285)\n\tat org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)\n\tat org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)\n\tat org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"
+        }
+      }
+    ],
+    "generated": "18:08:58 UTC"
+  }
+}

nifi/poc/responses/cluster-summary.json

@@ -0,0 +1,8 @@
+{
+  "clusterSummary": {
+    "connectedNodeCount": 0,
+    "totalNodeCount": 0,
+    "connectedToCluster": false,
+    "clustered": false
+  }
+}

nifi/poc/responses/flow-status.json

@@ -0,0 +1,20 @@
+{
+  "controllerStatus": {
+    "activeThreadCount": 0,
+    "terminatedThreadCount": 0,
+    "queued": "0 / 0 bytes",
+    "flowFilesQueued": 0,
+    "bytesQueued": 0,
+    "runningCount": 4,
+    "stoppedCount": 0,
+    "invalidCount": 0,
+    "disabledCount": 0,
+    "activeRemotePortCount": 0,
+    "inactiveRemotePortCount": 0,
+    "upToDateCount": 0,
+    "locallyModifiedCount": 0,
+    "staleCount": 0,
+    "locallyModifiedAndStaleCount": 0,
+    "syncFailureCount": 0
+  }
+}

nifi/poc/responses/jmx-metrics.json

@@ -0,0 +1,3 @@
+{
+  "jmxMetricsResults": []
+}