Support rotated certificate sources

[jroper]

Right now, there is a manual step involved when rotating certificates, of copying the certs. I think this is a feature gap, especially given that the sister project, cert-manager, does rotation automatically, and so there's not much more that needs to be done to implement completely automatic certificate management in a Kubernetes cluster.

What I'm thinking is allowing users to configure something like this in their Bundle configuration:

  sources:
  - secret:
      name: "some-cert-manager-issued-ca-cert"
      key: "tls.crt"
      keepCertHistory: true
      certHistoryLimit: 3

When trust-manager sees keepCertHistory: true, it will copy the cert itself into either a list of certs in the status of the Bundle CRD, or into a new CRD that exists for the purpose of storing historical certs. And any time it notices that the cert has changed, it will copy the new cert into the list (or create a new instance of the historical cert CRD), keeping the old cert around. The trust bundle will then contain both the current cert, and all previous certs that the operator has seen. The operator can clean old certs up, primarily this would be done based on the expiry date in the cert itself, but to prevent resource leaks (eg, in the case of misconfiguration of rotation), certHistoryLimit will specify the maximum number of old certs to keep, with certs being removed in an oldest first fashion if that number is ever reached. This should have a sensible non infinite default.

So, this will automate the copy step described in the docs, and provide a completely automated means for root CA rotation using cert-manager and trust-manager together.

[arsenalzp]

Hello,
Any suggestion to this request?

[erikgb]

I think this request will require a design before starting to implement anything.

[michaelw]

@jroper would it make sense to have certmgr create a cert backup (another labeled secret) on every successful rotation? That seems easier, allows per-original-cert policies wrt. number of backups, time periods, and is less prone to lost events. Then trustmgr merely aggregates these into a bundle (already supported, I believe, based on labels).

[arsenalzp]

Issue creator wants to have additional configmap with list of old/backed up certificates.
I'm wondering what are conditions to put certificates into backup list?

[jroper]

@michaelw That's another possibility. It does raise a number of issues - how do those certificates get named, for example?

Another issue that I see with that is potential pollution of the secrets resource - secrets are used for many, many different things. If someone does a kubectl get secrets, having histories of secrets outputted by cert-manager in there is going to, in some instances, make it harder to effectively manage secrets in there as there will now be a lot more "junk" that they need to sift through to find whatever secrets they are actually looking for. This different from say, a CertificateRequest resource, which also contains histories, because that resource is owned and only used by cert-manager, so nothing is being polluted when it keeps a history in there.

Another thing - the secrets are actually not what we want to keep around. It's only the cert/public key of the CA that we want to keep around. Discarding the old secret itself decreases the attack surface area that secret rotation is attempting to address. Keeping only the CA cert history in the Bundle status would be a more secure solution than keeping the private keys of all those old secrets around.

With regards to missed events - I think in typical configurations, root CAs are not rotated that frequently. I don't see a reason to rotate a root CA any more frequently than once a month. To miss that event, the trust manager operator would have to be not running for a month. And even without this feature, something is going to be going seriously wrong if the trust manager operator does not notice a root CA cert has changed, all the consumers of the bundle are going to stop trusting newly issued certificates, the behaviour will be no different whether this feature is implemented in cert-manager or trust-manager, the bundle will not contain the new certificate, it will only contain the old.

Also, expiration time periods is an intrinsic feature of certificates - it's already a per-original-cert configured feature, regardless of whether this is implemented in trust-manager or cert-manager. It would not make sense in my opinion to offer an additional time period anywhere, the thing cleaning them up would just clean them up on expiry.

[jroper]

@arsenalzp No, I'm not suggesting an additional configmap. I'm suggesting we have a new field in the status of the Bundle CRD, which contains a list of certificate histories. No configmap is involved at all.

The conditions for putting a new one in there are just if a new certificate is observed by the operator - every time it sees the certificate changed, it puts it in there. And then, there are two conditions for when they should be evicted - one, the certificate expires, two, the list of certificates in the history exceeds a certain number, in which case they are evicted in a FIFO fashion.

[michaelw]

@michaelw That's another possibility. It does raise a number of issues - how do those certificates get named, for example?

append unique suffix or version, e.g., similar to what helm does? If they are labeled, they are easy to select on as well. Upper limits are easy to configure, too.

Another issue that I see with that is potential pollution of the secrets resource

I see that, esp. with operators that list secrets (and their scalability issues). However, other parts of the eco system already do that, and it's even best practice in some places to not mutate secrets/CMs, but rather create new ones (and keep the old ones around for some time).

Another thing - the secrets are actually not what we want to keep around. It's only the cert/public key of the CA that we want to keep around.

yep, can be done, although perhaps configurable, for reasons of reverting changes?

Also, expiration time periods is an intrinsic feature of certificates - it's already a per-original-cert configured feature, regardless of whether this is implemented in trust-manager or cert-manager. It would not make sense in my opinion to offer an additional time period anywhere, the thing cleaning them up would just clean them up on expiry.

there are definitely use cases where we need a configurable grace period, i.e., CA gets rotated well before actual expiration, then is kept around while new leaf certs are already minted from the new CA, but both old and new CAs will validate certs up until some time when all clients have new leaf certs (which might still be well before expiration of the old CA).

[jroper]

it's even best practice in some places to not mutate secrets/CMs, but rather create new ones (and keep the old ones around for some time).

In a typical configuration, a CA certificate secret is then referenced by a CA Issuer or ClusterIssuer by name, and that issuer then issues leaf certificates. If cert-manager did an immutable rotation like this, by always generating new secrets with new names, it would break that mechanism. Either the rotation would have to update the CA Issuer that referenced the secret to have the new name of the new secret, or CA issuers would need to modified to have some mechanism to know how to dynamically work out what the current certificate is without referring to them by name, perhaps by referring to the Certificate resource rather than the Secret resource. I think making it possible for CA issuers to refer to Certificate resources rather than Secret resources would be a practical solution, the CA issuer could then read the status of Certificate to determine the current secret and read it from that, that would make sense in that instance, but otherwise, I don't think I think mutating the secret is the only practical way to continue supporting CA certificate issuers. So, the current secret would have to have a static name, and old secrets would need to be kept by copying the current secret into a new secret with a generated name.

there are definitely use cases where we need a configurable grace period

That grace period is already implemented, the CA already gets rotated well before actual expiration, that's the renewBefore field of Certificate. Now, yes, maybe new certs might be minted from the new CA before the old root cert expires, but that doesn't mean that you stop trusting the old root cert then. Everything is asynchronous. And just because a new certificate has been minted, doesn't mean everything has picked that new cert up yet. And there's no simple way to verify that everything using those certs has a) had a new cert minted, and b) has picked it up. If a certain application has a valid certificate to present for authentication, why would it reread it from its configuration source? There's no reason to, it has a certificate that it knows is valid, that's an intrinsic feature of TLS certificates, you can verify the validity of them statelessly, without needing to poll a central authority. If you remove the old CA from the bundle before everything has expired, you risk breaking things. It would be a bad practice to do so.

[michaelw]

it's even best practice in some places to not mutate secrets/CMs, but rather create new ones (and keep the old ones around for some time).

In a typical configuration, a CA certificate secret is then referenced by a CA Issuer or ClusterIssuer by name, and that issuer then issues leaf certificates. If cert-manager did an immutable rotation like this, by always generating new secrets with new names, it would break that mechanism.

yep, change would simply be too large. Going back to my initial suggestion, I can work with a CA rotation updating the associated secret, and also creating a new object (secret or otherwise), that holds the old CA cert, with configurable max number of versions kept around (really similar how helm configuration works). If that object is labelled appropriately, we could have trustmgr simply collect them into bundles (new CA, plus all N (non-expired?) previous CAs). that can then be distributed to consumers.

there are definitely use cases where we need a configurable grace period

That grace period is already implemented, the CA already gets rotated well before actual expiration, that's the renewBefore field of Certificate. Now, yes, maybe new certs might be minted from the new CA before the old root cert expires, but that doesn't mean that you stop trusting the old root cert then. Everything is asynchronous. And just because a new certificate has been minted, doesn't mean everything has picked that new cert up yet. And there's no simple way to verify that everything using those certs

We have that use case for mTLS, and it can be done, including forcing clients to renew certs on signal.

The real functionality I am after here for my use case is that after CA rotation (either forced or through nearing expiration), I have an automatic way to keep the old CA(s) in the trust anchor for as long as I need to, and can distribute them easily. My thinking was that If certmgr keeps the CA certs around after rotation, trustmgr simply has to combine them into a bundle (which it already can?). Would that shift the burden too much onto certmgr?

[jroper]

Would that shift the burden too much onto certmgr?

For me, it's not so much about what the burden is, it's more a property of what is logically responsible for this.

At a high level, cert-manager is responsible for key provisioning and distribution, while trust manager is responsible for trust distribution. Is keeping a history of certs a key provisioning and distribution concern, or a trust distribution concern? Because this history of certs is about what clients should trust, I'm inclined to consider it a trust distribution concern.

An interesting question to ask here is, what if the certs aren't provisioned by cert-manager, but by some other process, perhaps some cross k8s synchronisation mechanism is distributing these certs between kubernetes clusters in a multi-cluster service mesh, and you want trust manager to bundle the certs and distribute them. Does this feature still make sense then? I think it does, I can certainly imagine other scenarios where it would be really convenient for trust manager to keep the cert history and distribute it, even when not issuing the CA certs using cert-manager.

Conversely, what if trust-manager is not playing a part in bundling certs. Does it make sense for cert-manager to keep secret history? Are there any other use cases other than this trust manager use case for that? I don't think so.

So, to me, it makes much more sense to put this feature in trust-manager.

[arsenalzp]

@arsenalzp No, I'm not suggesting an additional configmap. I'm suggesting we have a new field in the status of the Bundle CRD, which contains a list of certificate histories. No configmap is involved at all.

The conditions for putting a new one in there are just if a new certificate is observed by the operator - every time it sees the certificate changed, it puts it in there. And then, there are two conditions for when they should be evicted - one, the certificate expires, two, the list of certificates in the history exceeds a certain number, in which case they are evicted in a FIFO fashion.

Hello,
Thank you for the explanation.
I meant what is the condition to recognize the changed certificate? From my point of view the changed certificate is just the brand new one.

[jroper]

Well, you've got the certificate history in the status, which includes the current cert. If the certificate from the secret is different to the current one from the certificate history, then you it's changed.

[anguslees]

Minor detail: we should keep all the CA certs that are valid (not expired or revoked yet). Not a "last N" history limit.

Rotating certs involves a period where multiple certs are valid. Each cert is either valid, or invalid/expired - the number of other valid/invalid certs doesn't change that, and counting them just introduces a concurrency challenge.

[jroper]

The default, as I've described it, is to keep all CA certs that are valid, and delete expired ones. But the problem with that is that if you set a high rotation rate (either intentionally, or due to misconfiguration), but have CA certs that are much longer lived than the rotation period, you can end up with a resource leak, where the certificate bundle becomes huge and this can have significant implications throughout your system. In circumstances where it's a misconfiguration, it may have the impact of turning an issue that would affect part of a system into an issue that brings the whole system down. Robust production environments should always put upper limits on the number of things that it will create, nothing should ever be allowed to grow to infinitely. That's the purpose of the certHistoryLimit, it eliminates the possibility of resource leaks.

I don't see the concurrency challenge. These certs have one owner - an operator. They are only ever created, updated or deleted by that owner. And operators run with a single writer principle. If you have a single owner that has an enforced single writer deployment, where's the concurrency challenge in that?