Ensure all state is shared across NiFi 2 cluster members

[Maleware]

Part of #809.

Why?

This opens the door for fixing the Oauth2 implementation which solves problems for many (including us).

Current state

Although it is termed a NiFi cluster, cluster members are not exactly replicas, and hold local state.

One example of local state is a Site-to-Site flowfile transfer to another cluster. Currently, to get the status of a transfer, you must connect to the node that you initiated the transfer on.

This might be the reason for the following:

• A JWT issuer per cluster memeber (tokens cannot be used on other cluster members)
• SNI hostname checks

Future state

All cluster state should be shared to cluster members so that cluster members can actually be replicas, and therefore sit behind a load balancer without needing sticky-sessions.

Recommended approach

• Try to find any reason why some state cannot be shared across cluster members.
• Start a conversation with NiFi maintainers to gauge interest in accepting this fix upstream.
• Fork NiFi and try to share the currently unshared state across cluster members.

Unknowns

• Which other state is local only? Perhaps the answer can be found in NIFI-7246

[NickLarsenNZ]

Is there a way to find out what state is not shared?