control: Ensure endpoints are driven to readiness (#1014)

When there are multiple replicas of a controller--especially the
destination controller--the proxy creates a load balancer to distribute
requests across all controller pods.

linkerd/linkerd2#6146 describes a situation where controller connections
fail to be established because the client stalls for 50s+ between
initiating a connection and sending a TLS ClientHello, long after the
server has timed out the idle connection.

As it turns out, the controller client does not necessarily drive all of
its endpoints to readiness. Because load balancers are designed to
process requests when only a subset of endpoints are available, the load
balancer cannot be responsible for driving all endpoints in a service to
readiness and we need a `SpawnReady` layer that is responsible for
driving individual endpoints to readiness. While the outbound proxy's
balancers are instrumented with this layer, the controller clients were
not configured this way when load balancers were introduced.

We likely have not encountered this previously because the balancer
should effectively hide this problem in most cases: as long as a single
endpoint is available requests should be processed as expected; and if
there are no endpoints available, the balancer would drive at least one
to readiness in order to process requests.

Author: olix0r

linkerd/app/core/src/control.rs

@@ -93,6 +93,9 @@ impl Config {
             .push_timeout(self.connect.timeout)
             .push(self::client::layer())
             .push(reconnect::layer(connect_backoff))
+            // Ensure individual endpoints are driven to readiness so that the balancer need not
+            // drive them all directly.
+            .push_on_response(svc::layer::mk(svc::SpawnReady::new))
             .push(self::resolve::layer(dns, resolve_backoff))
             .push_on_response(self::control::balance::layer())
             .into_new_service()