reverse tunnels: allow data plane to have tunnels to n control planes on the same upstream cluster

[AntonKanug]

We want to have multiple control planes for higher availability behind one upstream cluster.

An option:
In the control plane's reverse tunnel filter, add an option to only allow data plane tunnels intended for this control plane. Hoping that with data plane's retries and eventually the intended number of tunnels to the correct control plane will be created (with an appropriate load balancing policy).

Currently no information the intended control plane is sent to control plane (no headers), this will need to be added.

Happy to contribute this

[AntonKanug]

cc @agrawroh

[agrawroh]

@AntonKanug Are you thinking of a scenario where we have a Load Balancer in between the Data Plane and the Control Plane and there are multiple Responder Envoy replicas in CP?

For example, Initiator (Data Plane) => LB => 3x Responders (Control Plane)

I think having a config as part of the responder on how tunnels it should accept from a given Initiator should be able to solve this. In that case we could keep returning a configured status code if the connection lands on the same responder and eventually the initiator should be able to have connections with the intended number of responders.

For example, assume that initiator need 6 connections and each responder limits the incoming connections from each initiator to only 2. Then initiator will eventually be able to connect to all three responders after some retries.

WDYT?

[AntonKanug]

@agrawroh yeah the setups is Initiator (Data Plane) => envoy reverse proxy => 3x Responders (Control Plane)

initiator need 6 connections and each responder limits the incoming connections from each initiator to only 2

yup this approach makes sense to me, this way we dont need changes on the dataplane side (sending extra headers)

I think having a config as part of the responder

what does this look like? something like max_tunnels_per_node: int with a default to maxint (current behaviour)

[AntonKanug]

One potential issue with max tunnels approach on the control plane is that you couldnt increase number of tunnels for just one data plane since you would have to then bump the max tunnels on control plane. Which then leads to other data planes with less required tunnels not being distributed evenly.