Application pods fail to contact cluster services

[mdnfiras]

this right here (from the linkerd faq) says that:

The proxies are designed to continue operating even if they can’t reach the control plane. If the control plane dies, existing proxies will continue to operate with the latest service discovery information.

Is the control plane mentioned in the above text is the linkerd control plane? because in such case, it doesn't seem to be true:

linkerd control plane available, running wget from a pod to:

• kubernetes api ✅

/ # wget --no-check-certificate -S https://kubernetes
Connecting to kubernetes (10.96.0.1:443)
  HTTP/1.1 403 Forbidden
wget: server returned error: HTTP/1.1 403 Forbidden

• some other nginx pod ✅

/ # wget --no-check-certificate -S nginx.nginx
Connecting to nginx.nginx (10.105.252.189:80)
  HTTP/1.1 200 OK
  date: Wed, 29 Jun 2022 12:15:09 GMT
  content-length: 0
  
saving to 'index.html'
'index.html' saved

• google.com ✅

/ # wget --no-check-certificate -S https://google.com
Connecting to google.com (142.251.37.238:443)
  HTTP/1.1 301 Moved Permanently
  Location: https://www.google.com/
Connecting to www.google.com (172.217.171.196:443)
  HTTP/1.1 200 OK
  Date: Wed, 29 Jun 2022 12:14:18 GMT
  Expires: -1
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=ISO-8859-1
  P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
  Server: gws
  X-XSS-Protection: 0
  X-Frame-Options: SAMEORIGIN
  Set-Cookie: 1P_JAR=2022-06-29-12; expires=Fri, 29-Jul-2022 12:14:18 GMT; path=/; domain=.google.com; Secure
  Set-Cookie: AEC=AakniGOo2fm3dCCuPZTiftK0LGir8lIoiBDEhqceig6Y6Co-MV69769VPA; expires=Mon, 26-Dec-2022 12:14:18 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
  Set-Cookie: NID=511=gayYVDyLj2V49XJl0JC2bixweY0OBtbUa4-dyiJnQNP1ru6mm-SN9AzuMsnk7GhGLAd0-TcfJDsrcNqlkhAD2Jl8qGh1WT9_6V_95i7J3oNdsZVbBaBYCCXNYhPuL0CWi2n23d9xntDS4YI3YjU0YvcO7kk1ykPFYGq8MiZ9GyI; expires=Thu, 29-Dec-2022 12:14:18 GMT; path=/; domain=.google.com; HttpOnly
  Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
  Accept-Ranges: none
  Vary: Accept-Encoding
  Connection: close
  Transfer-Encoding: chunked
  
saving to 'index.html'
index.html           100% |******************************************************************************************************************************************| 16287  0:00:00 ETA
'index.html' saved

linkerd control plane unavailable (linkerd-destination deployment scaled down to 0), running wget from a pod to:

• kubernetes api ❌

/ # wget --no-check-certificate -S https://kubernetes
Connecting to kubernetes (10.96.0.1:443)
wget: short read, have only 0: Connection reset by peer
wget: error getting response: Connection reset by peer

• some other nginx pod ❌

/ # wget --no-check-certificate -S nginx.nginx
Connecting to nginx.nginx (10.105.252.189:80)
wget: error getting response: Connection reset by peer

• google.com ✅

/ # wget --no-check-certificate -S https://google.com
Connecting to google.com (142.251.37.238:443)
  HTTP/1.1 301 Moved Permanently
  Location: https://www.google.com/
Connecting to www.google.com (172.217.171.196:443)
  HTTP/1.1 200 OK
  Date: Wed, 29 Jun 2022 12:21:12 GMT
  Expires: -1
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=ISO-8859-1
  P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
  Server: gws
  X-XSS-Protection: 0
  X-Frame-Options: SAMEORIGIN
  Set-Cookie: 1P_JAR=2022-06-29-12; expires=Fri, 29-Jul-2022 12:21:12 GMT; path=/; domain=.google.com; Secure
  Set-Cookie: AEC=AakniGOsLlt5GM3t5ODR4QID-9RRv6XkHzSSCX-jW8lnu6rE9gA7jtQUig; expires=Mon, 26-Dec-2022 12:21:12 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
  Set-Cookie: NID=511=mEdB58x7k7JRIijyuV1LiOAfsi-6JLVxR4UB_L7RKb0kY3V8OxpvWaFLfq3FQ5TfQ8A0gfbwUYGpga3vSGqp6H6qXQtq6zpHiQdxd4mmJrp4dBUbMS2pEilnDbmu0thv1fcUZqWYDIXFX8vGmYOKgJz-8BeScw-utOUNmEaoqkI; expires=Thu, 29-Dec-2022 12:21:11 GMT; path=/; domain=.google.com; HttpOnly
  Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
  Accept-Ranges: none
  Vary: Accept-Encoding
  Connection: close
  Transfer-Encoding: chunked
  
saving to 'index.html'
index.html           100% |******************************************************************************************************************************************| 16291  0:00:00 ETA
'index.html' saved

so if linkerd control plane is down, all communication to internal cluster services won't work. is this intended?

[mateiidavid]

Hey @mdnfiras,

The intended functionality is substantiated in the first quote you provided:

The proxies are designed to continue operating even if they can’t reach the control plane. If the control plane dies, existing proxies will continue to operate with the latest service discovery information.

This refers to scenarios where connections have already been established. Consider, for example, a client pod (e.g a load testing tool) and a server pod (an http service we want to load test). The control plane has to be up and running for initial service discovery to be performed. The proxy will also need to be initialized by acquiring identity, which means when a pod first comes up the control plane must be functioning properly. Once the client starts sending traffic to the target, the proxy will perform service discovery: it collects metadata and the endpoints of the target by querying the control plane. If the connections are established and the control plane goes down, everything will continue to work as long as:

• Connections do not get terminated and (/or) idle out. In such cases, it's possible the service discovery information is dropped from the cache so subsequent connections might fail.
• Pods do not change their state (e.g server pods going down in which case the proxy won't be able to forward traffic, for obvious reason).

In your example, you are using an injected wget pod to send traffic to external services. The proxy will not perform service discovery on any of them, it will just transparently forward everything through. You also seem to be establishing connections to internal cluster services after the control plane is down.

Hope this answers your question!

[mdnfiras]

@mateiidavid thanks a lot for the quick response!

perhaps it would be nice to clarify this in that FAQ section, that this refers to scenarios where connections have already been established to avoid future confusion.

thanks again for your time, it is very much appreciated :D