Skip TLS and H2 when target is inbound IP (#1219)

In order to prevent the proxy from forwarding inbound traffic to target
IP addresses not in the list of inbound ips allowed by a pod, the proxy
will be changed to also forward on the original destination rather than
the original port on localhost. To support forwarding on the original
destination address, an iptables rule that currently results in a
traffic loop has to be dropped. The rule allows an app to call itself:
when a packet is sent over lo as a network interface and the destination
is not localhost, the packet is routed back through the inbound chain
(essentially resulting in the following flow: appX -- outbound --
inbound -- appX). The packets sent by the proxy are routed by the kernel
through the loopback interface, however, when forwarding on the original
destination, the destination address is no longer localhost (resulting
in a traffic).

Dropping the rule will break the edge case of an app calling itself
since the packet will go through the outbound and from outbound straight
to the process (packets on loopback skip nat prerouting tables so from
outbound /it has/ to be redirected through rules to the inbound; inbound
will not pick up the packet) -- the packet will be encrypted by the
outbound side and also upgraded to H2 since the outbound does not know
the receving end will not be another proxy.

This change deals with this edge case: if our target destination is also
part of the inbound ips then we will not do any TLS or upgrade the
connection to H2. Packet generated by the outbound side of the proxy
will now be sent straight to the application process who will be able to
deal with it as if it came from the inbound side. To support this
change, the list of inbound ips has been wired through the outbound
configuration. On each endpoint, we cross check the target against our
list of inbound ips -- if the target is an inbound ip, then we don't do
any TLS (reason: loopback) and set the protocol hint as Unknown. This is
done on both logical and direct connections; skipping TLS should be
protocol agnostic.

The gateway also had to be changed since it now supports logical and
direct connections. Where the configuration is not wired through, we
provide defaults (an empty set should still set TLS).

N.B. we set this directly on an endpoint for direct communications
however iptables currently has a rule which will forward a packet
directly to the application process when the address used is the
endpoint address (effectively skipping outbound). App X (ep addres) -->
App X, App X (logical address) --> outbound --> App X.

Signed-off-by: Matei David <[email protected]>

Author: mateiidavid

linkerd/app/gateway/src/gateway.rs

@@ -69,6 +69,8 @@ where
         // Create an outbound target using the endpoint from the profile.
         if let Some((addr, metadata)) = profile.endpoint() {
             debug!("Creating outbound endpoint");
+            // Create empty list of inbound ips, TLS shouldn't be skipped in
+            // this case.
             let svc = self
                 .outbound
                 .new_service(svc::Either::B(outbound::http::Endpoint::from((
@@ -78,6 +80,9 @@ where
                         metadata,
                         tls::NoClientTls::NotProvidedByServiceDiscovery,
                         profile.is_opaque_protocol(),
+                        // Address would not be a local IP so always treat
+                        // target as remote in this case.
+                        &Default::default(),
                     ),
                 ))));
             return Gateway::new(svc, http.target, local_id);

linkerd/app/gateway/src/lib.rs

@@ -117,6 +117,7 @@ where
         .push_tcp_endpoint()
         .push_tcp_forward()
         .into_stack();
+    let inbound_ips = outbound.config().inbound_ips.clone();
     let tcp = endpoint
         .push_switch(
             move |(profile, _): (Option<profiles::Receiver>, _)| -> Result<_, Error> {
@@ -130,6 +131,7 @@ where
                         metadata,
                         tls::NoClientTls::NotProvidedByServiceDiscovery,
                         profile.is_opaque_protocol(),
+                        &inbound_ips,
                     )));
                 }
 

linkerd/app/outbound/src/endpoint.rs

@@ -7,7 +7,12 @@ use linkerd_app_core::{
     transport::{self, addrs::*},
     transport_header, Conditional,
 };
-use std::{fmt, net::SocketAddr};
+use std::{
+    collections::HashSet,
+    fmt,
+    net::{IpAddr, SocketAddr},
+    sync::Arc,
+};
 
 #[derive(Clone, Debug)]
 pub struct Endpoint<P> {
@@ -19,9 +24,10 @@ pub struct Endpoint<P> {
     pub opaque_protocol: bool,
 }
 
-#[derive(Copy, Clone)]
+#[derive(Clone)]
 pub struct FromMetadata {
     pub identity_disabled: bool,
+    pub inbound_ips: Arc<HashSet<IpAddr>>,
 }
 
 // === impl Endpoint ===
@@ -40,13 +46,23 @@ impl Endpoint<()> {
 
     pub fn from_metadata(
         addr: impl Into<SocketAddr>,
-        metadata: Metadata,
+        mut metadata: Metadata,
         reason: tls::NoClientTls,
         opaque_protocol: bool,
+        inbound_ips: &HashSet<IpAddr>,
     ) -> Self {
+        let addr: SocketAddr = addr.into();
+        let tls = if inbound_ips.contains(&addr.ip()) {
+            metadata.clear_upgrade();
+            tracing::debug!(%addr, ?metadata, ?addr, ?inbound_ips, "Target is local");
+            tls::ConditionalClientTls::None(tls::NoClientTls::Loopback)
+        } else {
+            FromMetadata::client_tls(&metadata, reason)
+        };
+
         Self {
-            addr: Remote(ServerAddr(addr.into())),
-            tls: FromMetadata::client_tls(&metadata, reason),
+            addr: Remote(ServerAddr(addr)),
+            tls,
             metadata,
             logical_addr: None,
             opaque_protocol,
@@ -131,7 +147,6 @@ impl<P: std::hash::Hash> std::hash::Hash for Endpoint<P> {
 }
 
 // === EndpointFromMetadata ===
-
 impl FromMetadata {
     fn client_tls(metadata: &Metadata, reason: tls::NoClientTls) -> tls::ConditionalClientTls {
         // If we're transporting an opaque protocol OR we're communicating with
@@ -166,11 +181,18 @@ impl<P: Copy + std::fmt::Debug> MapEndpoint<Concrete<P>, Metadata> for FromMetad
         &self,
         concrete: &Concrete<P>,
         addr: SocketAddr,
-        metadata: Metadata,
+        mut metadata: Metadata,
     ) -> Self::Out {
         tracing::trace!(%addr, ?metadata, ?concrete, "Resolved endpoint");
-        let tls = if self.identity_disabled {
-            tls::ConditionalClientTls::None(tls::NoClientTls::Disabled)
+        let tls = if self.identity_disabled || self.inbound_ips.contains(&addr.ip()) {
+            let reason = if self.identity_disabled {
+                tls::NoClientTls::Disabled
+            } else {
+                metadata.clear_upgrade();
+                tracing::debug!(%addr, ?metadata, ?addr, ?self.inbound_ips, "Target is local");
+                tls::NoClientTls::Loopback
+            };
+            tls::ConditionalClientTls::None(reason)
         } else {
             Self::client_tls(&metadata, tls::NoClientTls::NotProvidedByServiceDiscovery)
         };

linkerd/app/outbound/src/http/logical.rs

@@ -45,7 +45,13 @@ impl<E> Outbound<E> {
                 .check_service::<ConcreteAddr>()
                 .push_request_filter(|c: Concrete| Ok::<_, Infallible>(c.resolve))
                 .push(svc::layer::mk(move |inner| {
-                    map_endpoint::Resolve::new(endpoint::FromMetadata { identity_disabled }, inner)
+                    map_endpoint::Resolve::new(
+                        endpoint::FromMetadata {
+                            identity_disabled,
+                            inbound_ips: config.inbound_ips.clone(),
+                        },
+                        inner,
+                    )
                 }))
                 .check_service::<Concrete>()
                 .into_inner();

linkerd/app/outbound/src/lib.rs

@@ -30,7 +30,13 @@ use linkerd_app_core::{
     transport::{self, addrs::*},
     AddrMatch, Error, ProxyRuntime,
 };
-use std::{collections::HashMap, fmt::Debug, time::Duration};
+use std::{
+    collections::{HashMap, HashSet},
+    fmt::Debug,
+    net::IpAddr,
+    sync::Arc,
+    time::Duration,
+};
 use tracing::info;
 
 const EWMA_DEFAULT_RTT: Duration = Duration::from_millis(30);
@@ -45,6 +51,7 @@ pub struct Config {
     // not perform per-target-address discovery. Non-HTTP connections are
     // forwarded without discovery/routing/mTLS.
     pub ingress_mode: bool,
+    pub inbound_ips: Arc<HashSet<IpAddr>>,
 }
 
 #[derive(Clone, Debug)]

linkerd/app/outbound/src/switch_logical.rs

@@ -26,7 +26,8 @@ impl<S> Outbound<S> {
         SSvc::Future: Send,
     {
         let no_tls_reason = self.no_tls_reason();
-        self.map_stack(|_, _, endpoint| {
+        self.map_stack(|config, _, endpoint| {
+            let inbound_ips = config.inbound_ips.clone();
             endpoint
                 .push_switch(
                     move |(profile, target): (Option<profiles::Receiver>, T)| -> Result<_, Infallible> {
@@ -39,6 +40,7 @@ impl<S> Outbound<S> {
                                     metadata,
                                     no_tls_reason,
                                     rx.is_opaque_protocol(),
+                                    &*inbound_ips,
                                 )));
                             }
 

linkerd/app/outbound/src/tcp/logical.rs

@@ -53,7 +53,13 @@ where
                 .check_service::<ConcreteAddr>()
                 .push_request_filter(|c: Concrete| Ok::<_, Infallible>(c.resolve))
                 .push(svc::layer::mk(move |inner| {
-                    map_endpoint::Resolve::new(endpoint::FromMetadata { identity_disabled }, inner)
+                    map_endpoint::Resolve::new(
+                        endpoint::FromMetadata {
+                            identity_disabled,
+                            inbound_ips: config.inbound_ips.clone(),
+                        },
+                        inner,
+                    )
                 }))
                 .check_service::<Concrete>()
                 .into_inner();

linkerd/app/outbound/src/tcp/opaque_transport.rs

@@ -160,6 +160,7 @@ mod test {
             metadata,
             tls::NoClientTls::NotProvidedByServiceDiscovery,
             false,
+            &Default::default(),
         )
     }
 

linkerd/app/outbound/src/test_util.rs

@@ -43,6 +43,7 @@ pub fn default_config() -> Config {
             max_in_flight_requests: 10_000,
             detect_protocol_timeout: Duration::from_secs(3),
         },
+        inbound_ips: Default::default(),
     }
 }
 

linkerd/app/src/env.rs

@@ -12,13 +12,13 @@ use inbound::policy;
 use std::{
     collections::{HashMap, HashSet},
     fs,
-    net::SocketAddr,
+    net::{IpAddr, SocketAddr},
     path::PathBuf,
     str::FromStr,
     time::Duration,
 };
 use thiserror::Error;
-use tracing::{error, info, warn};
+use tracing::{debug, error, info, warn};
 
 /// The strings used to build a configuration.
 pub trait Strings {
@@ -68,6 +68,12 @@ pub enum ParseError {
     NotANetwork,
     #[error("host is not an IP address")]
     HostIsNotAnIpAddress,
+    #[error("not a valid IP address: {0}")]
+    NotAnIp(
+        #[from]
+        #[source]
+        std::net::AddrParseError,
+    ),
     #[error(transparent)]
     AddrError(addr::Error),
     #[error("not a valid identity name")]
@@ -168,6 +174,8 @@ pub const ENV_POLICY_SVC_BASE: &str = "LINKERD2_PROXY_POLICY_SVC";
 pub const ENV_POLICY_WORKLOAD: &str = "LINKERD2_PROXY_POLICY_WORKLOAD";
 pub const ENV_POLICY_CLUSTER_NETWORKS: &str = "LINKERD2_PROXY_POLICY_CLUSTER_NETWORKS";
 
+pub const ENV_INBOUND_IPS: &str = "LINKERD2_PROXY_INBOUND_IPS";
+
 pub const ENV_IDENTITY_DISABLED: &str = "LINKERD2_PROXY_IDENTITY_DISABLED";
 pub const ENV_IDENTITY_DIR: &str = "LINKERD2_PROXY_IDENTITY_DIR";
 pub const ENV_IDENTITY_TRUST_ANCHORS: &str = "LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS";
@@ -390,6 +398,19 @@ pub fn parse_config<S: Strings>(strings: &S) -> Result<super::Config, EnvError>
         .unwrap_or_else(|| parse_dns_suffixes(DEFAULT_DESTINATION_PROFILE_SUFFIXES).unwrap());
     let dst_profile_networks = dst_profile_networks?.unwrap_or_default();
 
+    let inbound_ips = {
+        let ips = parse(strings, ENV_INBOUND_IPS, parse_ip_set)?.unwrap_or_default();
+        if ips.is_empty() {
+            info!(
+                "`{}` allowlist not configured, allowing all target addresses",
+                ENV_INBOUND_IPS
+            );
+        } else {
+            debug!(allowed = ?ips, "Only allowing connections targeting `{}`", ENV_INBOUND_IPS);
+        }
+        ips.into()
+    };
+
     let outbound = {
         let ingress_mode = parse(strings, ENV_INGRESS_MODE, parse_bool)?.unwrap_or(false);
 
@@ -441,6 +462,7 @@ pub fn parse_config<S: Strings>(strings: &S) -> Result<super::Config, EnvError>
                     .unwrap_or(DEFAULT_OUTBOUND_MAX_IN_FLIGHT),
                 detect_protocol_timeout,
             },
+            inbound_ips,
         }
     };
 
@@ -915,6 +937,12 @@ fn parse_socket_addr(s: &str) -> Result<SocketAddr, ParseError> {
     }
 }
 
+fn parse_ip_set(s: &str) -> Result<HashSet<IpAddr>, ParseError> {
+    s.split(',')
+        .map(|s| s.parse::<IpAddr>().map_err(Into::into))
+        .collect()
+}
+
 fn parse_addr(s: &str) -> Result<Addr, ParseError> {
     Addr::from_str(s).map_err(|e| {
         error!("Not a valid address: {}", s);