tls: Fix inbound I/O when TLS detection fails (#958)

When the proxy performs inbound TLS detection, it may break TLS streams
for application-terminated TLS connections.

Typically, inbound TLS detection does not actually need to buffer data
from the socket to determine the SNI value for a connection. However,
non-proxy clients may send enough ClientHello extensions such that
detection cannot be completed from a single peek of 512B. In this
situation, the proxy buffers data from the socket to determine the
SNI value, but it unfortunately does not preserve this buffered data as
it forward the connection.

This change updates the TLS detection logic to properly preserve any
data buffered from the socket in forwarded connections.

Author: olix0r

linkerd/tls/src/server/mod.rs

@@ -247,12 +247,14 @@ where
             Ok(Some(sni)) => {
                 trace!(%sni, "Identified non-matching SNI via peek");
                 let tls = Conditional::Some(ServerTls::Passthru { sni });
-                return Ok((tls, EitherIo::Left(io.into())));
+                let io = PrefixedIo::new(buf.freeze(), io);
+                return Ok((tls, EitherIo::Left(io)));
             }
 
             Ok(None) => {
                 trace!("Not a matching TLS ClientHello");
-                return Ok((NO_TLS_META, EitherIo::Left(io.into())));
+                let io = PrefixedIo::new(buf.freeze(), io);
+                return Ok((NO_TLS_META, EitherIo::Left(io)));
             }
 
             Err(client_hello::Incomplete) => {