New Linkerd Configuration Management

[Pothulapati]

Problem

Currently, State management and CLI Rendering code in Linkerd is complicated as there are two different types i.e installOptions (i.e the CLI flags) and Values (which are used to render the manifests) and even though Values seems like our main source of truth (as installOptions are converted into Values), we currently store the installOptions in linkerd-config (as the whole Values and Helm support came later). This requires multiple conversions between those types and makes adding new configuration options complicated.

Solution

The main aim of the solution is to store overridden Values directly in the cluster, so that installOptions will only be the face of the CLI and Values will be the single source of truth to be stored in the cluster. For upgrades, we can override the default Values (part of the CLI) with the ones stored in the cluster and use that for rendering.

This also eliminates existing different config fields i.e Global, Proxy, and Install in linkerd-config as Values contains everything in YAML.

Storage Format

As these Values contain sensitive data i.e private keys, etc, we must store them as a secret. Though the sensitive data can be split out into multiple secrets, allowing us to store this as a configMap. But that would mean that the CLI now has to fetch multiple things from the cluster and making it confusing for users, and hence we went with storing the Values as a secret, that we retrieve and directly use for upgrades. See #4911

Current dependencies on linkerd-config

Most control-plane components read configuration from linkerd-config. This dependency will be removed, and all the required configuration will be passed to all the components through flags. See #4915

Linkerd Check

The initial plan was to use the same secret to understand the current cluster-state and perform checks based on that. See #4941

But this means that Linkerd check will not work with Helm, as there is no overrides config secret in Helm, as Helm has its way of storing and using state. Storing the cluster state in a secret, also means that the current Linkerd state is a black-box for users.

The solution to this problem could be to have another resource that stores the full configuration to be present both with the CLI and Helm for check and to display config related purposes (linkerd config maybe?)

The new resource can be in one of the following formats:

• ConfigMap: Strip out all the sensitive data i.e keys and store that in the configMap to be used by linkerd check. Also, Helps users directly understand the state of the cluster by looking at it and should work both with Helm and CLI.
• Secret: Store everything as a secret. Linkerd check will work as usual but linkerd config will have to strip out those secrets before rendering config to users.

The above-mentioned resource will be a template to be rendered directly (unlike the linkerd-overrides as it is needed only with CLI) both with Helm and CLI (as it is needed by both), and there are functions to remove specific fields from a map in Helm to support the first case

[adleong]

To add some more details about the specific resources:

secret/linkerd-config-overrides

This secret contains the subset of the Values which have been overridden in this install. This includes issuer keys if they were supplied at install time or if they were generated by the CLI. This resource is not part of the Helm chart and is only rendered by the Linkerd CLI (it will not exist when Linkerd is installed with Helm). This resource is ONLY used by the linkerd upgrade command and is not read by any control plane components or other CLI commands.

secret/linkerd-config

This secret contains the full Values used to render this install. This includes issuer keys if they were supplied at install time or if they were generated by the CLI. This resource is part of the Helm chart and will be present for both CLI and Helm installs. This resource is read by the CLI to get runtime configuration such as the cluster domain, trust domain, and proxy configuration. The linkerd check command needs this information to validate the configuration. The linkerd inject --manual command needs the proxy configuration to do manual injection. This resource is ONLY used by the CLI and is not read by any control plane components.

configmap/linkerd-config

This deprecated configmap contains a Config protobuf struct. It will no longer be rendered in new installs. Upgrading from a Linkerd version that has this configmap will require reading this configmap.

Workflows

CLI Install

When intsalling with the CLI, Linkerd will load the default Values from Values.yaml. It will then apply overrides to these Values based on the install CLI flags. It will render the overrides to secret/linkerd-config-overrides so that future upgrades will know which values have been overridden. It will also render the full Values to secret/linkerd-config so that CLI commands such as check and inject can get the runtime configuration.

Helm install

The helm install will render the full Values to secret/linkerd-config so that CLI commands such as check and inject can get the runtime configuration. The secret/linkerd-config-overrides will not be rendered.

CLI Upgrade

When upgrading with the CLI, the default values will be loaded from Values.yaml. Then stored overrides will be loaded from secret/linkerd-config-overrides and updated from the CLI upgrade flags. The updated overrides will be saved to secret/linkerd-config-overrides and the updated full values will be saved to secret/linkerd-config.

Helm Upgrade

The regular Helm upgrade workflow.

FAQ

Why do the new resources need to be secrets?

The Values used to render the charts contain sensitive information such as the issuer key.

Why do we need to store the issuer key again?

Storing the full values and the values overrides allows us to greatly simplify our implementation. This allows the upgrade command to read the overridden values all at once from one place without needing to understand them rather than needing to additionally read the issuer secret. It also allows us to use the values as the source of truth for runtime configuration rather than needing to keep multiple representations in sync.

Will control plane components be able to read these new secrets?

No. No RBAC access to these secrets will be granted. Any configuration that the control plane components need will be passed to them as flags. Only the CLI will ever access these secrets.

Isn't this what Helm already does?

Yes, Helm install does store the full values and overridden values in a secret. We replicate this idea for use by our CLI.

Can we reuse Helm under the hood to avoid replicating this behavior?

As far as I have been able to find, the Helm code does not provide or expose a public interface to make this possible.