Support for running with Kubernetes as quorum manager instead of ZooKeeper

[soenkeliebau]

NiFi now supports running with Kubernetes as quorum manager instead of ZooKeeper. We should implement this and offer it as an option.

In principle this is just a few settings that could be overriden, but an entry needs to be written in an xml file, which we currently hard-code and effectively kill these overrides.

• Support Zookeeperless NiFi #775
• Use NiFi without Zookeeper; fix various problems demos#221
• test: Switch Iceberg test to not use Zookeeper #800

[nightkr]

Looks like this has two components:

1. Clustering coordination (https://nifi.apache.org/nifi-docs/administration-guide.html#kubernetes-clustering)

This just requires us to set nifi.cluster.leader.election.implementation, and give NiFi permission to manage Lease objects:

---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: simple-nifi
spec:
  # [snip]
  nodes:
    configOverrides:
      nifi.properties:
        nifi.cluster.leader.election.implementation: KubernetesLeaderElectionManager
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nifi-extra
rules:
  # https://nifi.apache.org/nifi-docs/administration-guide.html#kubernetes-clustering
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - create
      - get
      - update
      # undocumented but required
      - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: nifi-extra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nifi-extra
subjects:
  - kind: ServiceAccount
    name: simple-nifi-serviceaccount

(Though obviously we should provide proper support, this is just illustrative of the scope of change we'd need to make.)

2. Cluster state storage (https://nifi.apache.org/nifi-docs/administration-guide.html#kubernetes-configmap-cluster-state-provider)

This is configured by the XML file Sönke refers to in the OP. In theory the change scope should be similarly small, but this is indeed hard-coded and would require an operator change.

[sbernauer]

@soenkeliebau will have a look and try it out. @maltesander is up to also helping out

[nightkr]

Release note:

The Stackable Operator for NiFi now supports using Kubernetes for cluster coordination and storage, rather than ZooKeeper. (Requires NiFi 2.x, migrating existing installs is currently not supported.)

Docs: https://github.com/stackabletech/nifi-operator/blob/c6e3b2831a650153d5d3f293fb9e3b9f65439557/docs/modules/nifi/pages/usage_guide/clustering.adoc#backend-kubernetes (also reorganized the page to focus on clustering in general rather than ZooKeeper in specific.)