errors: always teardown connections on errors (#939)

Currently, the proxy tears down connections on unhandled errors, unless
those errors are timeouts. We special-case errors which are timeouts and
don't close connection in those cases.  However, this can result in
connections failing to be torn down when the host is unreachable. If the
proxy continues trying to reconnect after a connection error, the
connection error won't be what makes it to the errors layer --- instead,
the request will eventually time out while reconnecting. This gets the
request 503ed, but the connection remains open --- which, when a client
is trying to talk to a stale endpoint, results in the connection
entering a stuck state.

This branch fixes this by removing the special-case logic for timeouts,
and always tearing down connections. I validated this change using the
repro in linkerd/linkerd2#5871 and confirmed that the issue is resolved
after removing the special handling for timeouts.

This fixes linkerd/linkerd2#5871.

Author: hawkw

linkerd/app/core/src/errors.rs

@@ -191,11 +191,9 @@ impl<RspB: Default + hyper::body::HttpBody> respond::Respond<http::Response<RspB
                 }
 
                 // Gracefully teardown the server-side connection.
-                if should_teardown_connection(&*error) {
-                    if let Some(c) = self.close.as_ref() {
-                        debug!("Closing server-side connection");
-                        c.close();
-                    }
+                if let Some(c) = self.close.as_ref() {
+                    debug!("Closing server-side connection");
+                    c.close();
                 }
 
                 if self.is_grpc {
@@ -223,16 +221,6 @@ impl<RspB: Default + hyper::body::HttpBody> respond::Respond<http::Response<RspB
     }
 }
 
-fn should_teardown_connection(error: &(dyn std::error::Error + 'static)) -> bool {
-    if error.is::<ResponseTimeout>() || error.is::<tower::timeout::error::Elapsed>() {
-        false
-    } else if let Some(e) = error.source() {
-        should_teardown_connection(e)
-    } else {
-        true
-    }
-}
-
 fn http_status(error: &(dyn std::error::Error + 'static)) -> StatusCode {
     if let Some(HttpError { http, .. }) = error.downcast_ref::<HttpError>() {
         *http