How to deal with external TLS services

[sebfortier2288]

Hello,
I have a question regarding the recommended way to deal with external services:

We are currently running linkerd 2.11.1 (Will be upgrading soon to latest patch version) in our production cluster. The main thing we wanted to achieve with linkerd was mTLS communication between pods, which is working perfectly.
The application running in the cluster is interacting with multiple other services that are outside of the cluster. These services are mostly accessible through https. For calls to these services, we have no special configuration in-place.
Since linkerd have been deployed in production, we have been experiencing intermittent connection errors to these external services (~1% fo the calls).
We troubleshooted these errors and found that it was "caused" by the linkerd proxy shutting down the connection after his connect-timeout of 1sec. The real issue is somewhere else in the network, probably an intermittent latency, but linkerd is now showing it:
linkerd_app_core::serve: Connection closed error=connect timed out after 1s

There is a configuration to globally increase this connect-timeout, but since it is a global setting, we fear that it will add more latency because we have a lot of "garbage traffic" being blocked by network policies or our enterprise firewall.

The only way we see to get around this issue quickly is to add the target ports of these external services to the "outbound-ports-to-ignore" setting. This is tricky because it simply skip entirely the proxies for these calls, preventing us to use other linkerd features in the future. Also, if we were using the same port for calls inside the cluster, they would also be skipped.

Ideally, it would be nice to be able to configure this connect-timeout (or other config) based on the external service (IP or DNS) being called, or even based on in-cluster vs out-of-cluster calls. It would let us configure a higher value for service A which we know sometimes take longer to connect to than an in-cluster service.

What is the recommended way to deal with external TLS services? Should we configure something for these services (i.e. service profile)? Is skipping the proxy really the only way?

Thank you for your help!

[adleong]

Hi @sebfortier2288!

These are good questions. Unfortunately, there isn't any way to configure the connect timeout based on the destination right now. However, you can use the config.linkerd.io/proxy-outbound-connect-timeout annotation to configure the outbound connect timeout differently on specific workloads rather than as a cluster wide setting. This would potentially allow you to set a higher connect timeout on pods which you know talk to these external services. Similarly, the config.linkerd.io/skip-outbound-ports annotation can be used to configure only certain pods to skip the proxy for certain ports.

[sebfortier2288]

Hi, thanks for your answer!

This confirms my first impressions. We'll stick to the use of the skip-outbound-ports feature for now and we'll see if we can isolate the workloads calling these services.

Thanks again!