ingress: Add failfast to the forwarder (#1035)

The ingress-mode proxy's forwarding stack--used when a request does not
set the `l5d-dst-override` header--has no failfast implemetation. This
means that when a connection can't be obtained for the endpoint,
requests are buffered indefinitely.

This change adds a failfast layer so that these requests are failed
eagerly after 3s of unavailability, causing the serverside connection to
be dropped (so that the application client may re-resolve the endpoint).

This is really a temporary solution. We should probably avoid
implementing reconnection at all in this case so that connection errors
can be used in place of failfast errors.

Related to linkerd/linkerd2#6184

Author: olix0r

linkerd/app/outbound/src/ingress.rs

@@ -168,6 +168,11 @@ impl Outbound<svc::BoxNewHttp<http::Endpoint>> {
                     })),
                 },
                 http_endpoint
+                    .push_on_response(
+                        svc::layers()
+                            .push(svc::layer::mk(svc::SpawnReady::new))
+                            .push(svc::FailFast::layer("Ingress server", dispatch_timeout)),
+                    )
                     .instrument(|_: &_| info_span!("forward"))
                     .into_inner(),
             )