Linkerd Roadmap Update

[olix0r]

Hi folks,

While we regularly discuss Linkerd's upcoming Roadmap at our monthly community meeting, we have not done a great job of publishing the roadmap to GitHub. We're generally light on project management, so we don't have a particularly groomed set of milestones and projects (yet, though they will show up as planning progresses). Instead, let me start a discussion about what we're planning to work on over the next few releases.

stable-2.8.0: Multi-Cluster

We're in the final stages of preparing Linkerd stable-2.8.0 for release. This release focuses on introducing multi-cluster features to Linkerd, so you have tools to federate identity-aware service discovery between clusters. The docs were just merged today, and @grampelberg has been writing a series of blog posts about the design we've chosen. @zaharidichev did a massive amount of work on this, so be sure to give him a (socially-distant) 🙌 if you see him.

We've also been hard at work applying feedback from the 2.7 releases. We've fixed a number of dreaded HTTP 502 scenarios:

• Services with only one client that polled exactly every 60s could see spurious 502 (Prometheus scrape occasionally fails with "server returned HTTP status 502 Bad Gateway" in Linkerd 2.7.1 #4274). We fixed the cache-eviction logic to protect against this race condition (buffer: Move idle timeouts into the buffer linkerd2-proxy#502 -- funny coincidence).
• The buffering strategy introduced in edge-20.4.1 and promoted in stable-2.7.1 added contention, and latency, especially at high concurrency levels, which could cause 502s. We've fixed it.
• We've also gotten a bit better at identifying problems caused by iptables & nf_conntrack (Increase nf_conntrack_tcp_timeout_close_wait #4276). Expect some new debugging guides on the website soon!
• gRPC users should no longer see spurious 502 responses (Server Side Streaming gRPC Proper Status Code Not Making it to Client #4262). Connection failures will instead manifest as responses with a grpc-status of UNAVAILABLE, like common gRPC clients handle these failures.

Our heartfelt thanks goes to everyone who took the time to open issues with helpful repro steps or to help test fixes. Good bug reports are an invaluable contribution to the community, making production better for us all. Thanks especially to @byblakeorriver, @ericsuhong, @naseemkullah & @praseodym for their recent contributions!

Today's edge-20.5.2 is effectively our first Release Candidate for stable-2.8.0. Try it out, and keep the feedback coming!

stable-2.9.0: Total Mesh mTLS

As most of you know, Linkerd's current mTLS implementation is limited to meshed HTTP traffic (including HTTP/2 & gRPC). This limitation is in place because Linkerd's TCP proxy has been a simple pass-thru, usually to kube-proxy. In this release we are going to implement TCP load balancing in Linkerd's proxy, which will allow us to bypass kube-proxy entirely so that we can establish mTLS between all meshed pods for all meshed traffic.

Before that, though, it's time for us to upgrade the proxy's networking stack so that we can take advantage of new features in Rust (async/await) and updated dependency versions. @hawkw is leading our migration to Tokio 0.2, and we're nearly to the point of being able to test the upgraded version of the proxy. We're taking a close look at performance--especially latency overhead--as we do this migration. As it completes, we'll also start investigating supporting a multi-core runtime to support massively-concurrent use cases for people with the CPU to spend. But our goal is to continue adding absolutely minimal overhead for use cases that don't require it.

stable-2.10.0: Mandatory TLS by Default

With multi-cluster and total mTLS stable, we'll finally be ready to tackle authorization policies. There are many, many, many ways to approach this. Rather than trying to tackle some Big Conceptual problem, we're trying to target a very specific use case: Linkerd requires that all non-TLS traffic is documented.

Linkerd will already ensure that all meshed traffic is private between endpoints; but we can also validate that all ingress/egress traffic is TLS'd (even when the proxy doesn't terminate the TLS). The only thing we really can't do is ensure that Kubernetes' liveness & readiness probes are private... So we need to build a mechanism to discover these policies at inject-time so that the proxy can start up safely, rejecting all traffic that is insecure.

Once this is done, we can pursue supporting finer-grained access control.

And more

This roadmap isn't meant to cover everything we're going to work on.

Off the top of my head, we're going to continue to invest in making Linkerd's Prometheus data excellent to use and easier ot manage. @Pothulapati is working on a proposal to allow you to configure an alternate Prometheus instance, preventing the need to have a Prometheus instance in the Linkerd control plane. And @adleong and I have been working to update the metrics we export to Prometheus so that operators have flexibility to drop high-cardinality metrics, and so that the metrics are generally easier to approach without understanding the proxy's internals.

@alpeb and @kleimkuhler continue to invest in our integration test infrastructure so that we can reliably reproduce reported issues and ensure they don't recur. We have two Summer of Code students working on supporting ARM environments (with conformance tests!).

It's an active community, so expect new contributions to fill this out with all sorts of other improvements that help make production better.

Starting a discussion

But, this is a discussion, so we're open to feedback about all of this. Are we missing things that will make your life easier? (Sorry in advance: Linkerd can't cut your hair). Are there any features you'd like to get involved with?

Sound off in the comments below!

[evhfla]

What about circuit breaking support?

[olix0r]

@evhfla I think this would be a great feature! #2846 is a long-standing issue that is marked as still needing design. We'd love to have someone in the community get involved to help scope this issue down... I think we have a good handle on how to implement the feature in the proxy, but the configuration-side of it needs some work.

If this is something you'd like to try getting involved with, we'd be happy to discuss it in more detail!

[evhfla]

Sure, because that's the only thing keeping me from migrating from 1.x to 2.x.

[ericsuhong]

I also would like to mention adding support for using external CA for mTLS (with auto-rotation support). We use Linkerd almost exclusively for mTLS, and this would really help us in the future.

I believe @dwj300 started this discussion and wants this feature in the roadmap as well.

[fabio-s-franco]

It's great to see total mesh in the roadmap. Since our main interservice communication is done via amqp, linkerd's benefits have been very limited and it has given us more trouble than the benefits. Look forward to stable 2.9.0 and keep up the good work :)

[duke-arioch]

I am also doing a bit of AMQP and some other TCP protocols. Looks like 2.10.0 will have a lot of improvements for TCP.

[snnsnn]

Please can you also address how to handle authorization using open policy agent both in app and service level, something like https://istio.io/latest/blog/2021/better-external-authz/ and https://www.openpolicyagent.org/docs/latest/envoy-authorization/. What is the best way to go?