KubeadmConfig changes should be reconciled for machine pools, triggering instance recreation

[AndiDog]

What would you like to be added (User Story)?

As operator, I want KubeadmConfig.spec.{files,preKubeadmCommands,...} changes to have an effect on MachinePool-creates nodes, resulting in server instance recreation.

Detailed Description

Discussed in office hours 2023-06-14 (notes, main points copied into this issue below).

Situation: MachinePool manifest references AWSMachinePool and KubeadmConfig (a very regular machine pool config)

Expectation: Changing KubeadmConfig.spec.* should lead to recreating (“rolling”) nodes. With infra provider CAPA, nothing happens at the moment. Here's why.

• Problem 1: CAPI’s KubeadmConfigReconciler does not immediately update the bootstrap secret once KubeadmConfig.spec changes, but only once it rotates the bootstrap token (purpose: new machine pool-created nodes can join the cluster later on). This means several minutes of waiting for reconciliation.

  • Suggestion: Simple bug fix. @AndiDog has a draft implementation that always considers updating the secret, not only if the token must be refreshed. In the meantime, users can work around by creating a new KubeadmConfig object.

• Problem 2: CAPA (and likely all other infra providers) does not watch the bootstrap secret, so it cannot immediately react to KubeadmConfig.spec changes either.

  • @AndiDog Should it even directly watch the secret? What should the CAPI ⇔ CAPx contract be?
  • @fabriziopandini: Watching secrets can blow up memory [of the kubectl client]. Think of the UX and possible solutions first.
  • @CecileRobertMichon: Maybe change MachinePool.spec.template.spec.bootstrap.dataSecretName every time because that triggers reconciliation for the MachinePool object (machinepool_controller_phases.go code).
  • @sbueringer: For MachinePool support in ClusterClass we have to decide what the “ideal” way to rollout BootstrapConfig is

• Problem 3: The bootstrap secret contains both the “how to set up this server init data” (e.g. cloud-init / ignition) and the random bootstrap token by which nodes join the cluster. If only the token gets refreshed (DefaultTokenTTL is 15 minutes), we don’t want nodes to be recreated, since that would recreate all nodes every few minutes.

  • @AndiDog CAPA does not trigger a node rollout (AWS ASG instance refresh) right now (Updating MachinePool, AWSMachinePool, and KubeadmConfig resources does not trigger an ASG instanceRefresh cluster-api-provider-aws#4071) because it has no logic to detect when the bootstrap config changed. How can CAPA (or another infra provider) tell that something apart from the bootstrap token has changed? We have a CAPA PR (Trigger machine pool instance refresh also on user data change (unless it's only the bootstrap token) cluster-api-provider-aws#4245) that checks “is there a difference apart from only the bootstrap token value” but that is very hacky and format/provider-specific. Maybe a "checksum without bootstrap token" provided by CAPI could help the infra provider?
  • @CecileRobertMichon: Split bootstrap token vs. other init data?

Anything else you would like to add?

• CAPA-specific issue: Updating MachinePool, AWSMachinePool, and KubeadmConfig resources does not trigger an ASG instanceRefresh cluster-api-provider-aws#4071. The problem applies to all infra providers, though.
• This issue asks for mutable KubeadmConfig, while another proposal wants to make it immutable even for machine pool (breaking change): Make KubeadmConfigTemplate immutable #4910.

Label(s) to be applied

/kind feature
/area bootstrap
/area machinepool

[killianmuldoon]

/triage accepted

[fabriziopandini]

Frankly speaking, what I found confusing in the UX is that we are changing an external reference in place, and this is inconsistent with everything else in CAPI. Additionally, the reference to a bootstrap object is defined inside a machine template, while e.g. in MD/MS we have a similar API modeling but we use a bootstrap template instead of a bootstrap object.

However, IMO while designing a solution for problem 1, I think we should aim for a solution where changes to external objects should surface to the parent objects, no matter if user-driven (e.g rotating template references) or machine-driven (by changing the data secret name).

This approach - changing the parent object - is consistent with everything else in CAPI and provides a clean pattern for all the providers, and IMO this could address problem 2.

WRT to problem 3, It seems to me that what we are saying is going in the direction of each machine having its own bootstrap token so machine provisioning is not affected by changed to concurrent changes to the bootstrap object, and this could be somehow overlapping with @Jont828 work (see eg. #8828 (comment))

Last note, what we are deciding

[AndiDog]

This approach - changing the parent object - is consistent with everything else in CAPI and provides a clean pattern for all the providers, and IMO this could address problem 2.

But then the providers need to watch KubeadmConfig. Right now, most of them only watch their own types (e.g. AWSCluster) and Cluster. I think it's the only reasonable way to implement good machine pool related reconciliation, so I'm fine with that proposal. Then how does a provider find out when the bootstrap data secret got updated, since that's when the provider needs to work on its machine pool, for example CAPA updating the auto-scaling group's latest launch configuration, put in the latest user data, and roll out new nodes. We need to also avoid timing bugs – if providers reconcile too early, they still see the old content of the secret.

---

Here is one idea how to add such a contract with providers:

• Providers should not checksum the content of the bootstrap data (which my first hacky attempt did: Trigger machine pool instance refresh also on user data change (unless it's only the bootstrap token) cluster-api-provider-aws#4245), but instead CAPI should reason about the data and report whether something changed. Since the bootstrap token is currently part of that data, and gets rotated regularly (is this for security reasons?), the bootstrap data content changes all the time but does not necessarily have relevant changes such as a change in KubeadmConfig.spec.*.

  • Therefore, CAPI could store a KubeadmConfig.status.dataSecretPartialChecksum which hashes the cloud-init/ignition content, excluding the random bootstrap token value. It should first update the secret and then write that field.
  • We can also put the checksum into the secret's labels for comparison.
  • Also, there should be a KubeadmConfig.status.dataSecretFullChecksum including hashing of the token.

• A provider like CAPA can watch KubeadmConfig objects which are related to AWSMachinePool.

  • Once the KubeadmConfig.status.dataSecretPartialChecksum value changes as compared to the latest rollout (for CAPA, this could be compared to a tag on the launch configuration version), the controller applies the latest bootstrap data to the machine pool, rolling out new nodes (for CAPA, this means to create a new launch configuration version and also trigger instance refresh).
  • If only KubeadmConfig.status.dataSecretFullChecksum changes, but not KubeadmConfig.status.dataSecretPartialChecksum, it means the bootstrap token got rotated and the new bootstrap data should only apply to new nodes without rolling existing nodes (for CAPA, this means to create a new launch configuration version but not triggering instance refresh).

• Since this only adds new fields, this does not look like a breaking change for providers. They can migrate to this contract when they're ready. For backward compatibility to older CAPI versions, meaning the checksum fields are defaulted to "", providers can fall back to their old behavior.

What do you think of this?

[CecileRobertMichon]

But then the providers need to watch KubeadmConfig

This is a dealbreaker if we're talking about watching specific Kubeadm* types. Infra providers should be bootstrap provider agnostic and work with all bootstrap providers and control providers as kubeadm is not the only one out there. If we're trying to watch the bootstrap reference that would be fine as we can do that without referencing kubeadmconfig specifically. In that case, status.dataSecretPartialChecksum would need to become part of the bootstrap provider contract and every bootstrap provider would need to implement it.

[AndiDog]

@CecileRobertMichon I agree that infra providers shouldn't directly watch KubeadmConfig/Secret, but then how would AWSMachinePool reconciliation get triggered immediately? Having CAPA reconcile every few minutes, as it does right now (implementation detail!), isn't really enough: it sees that the secret has changed and can react as we desire, but as a user I really expect this to happen within seconds.

The hierarchy is MachinePool > KubeadmConfig > Secret, so if the AWSMachinePool controller should get triggered, we'd need to write something into at least MachinePool.status, I guess.

CAPA's current controller has this:

func (r *AWSMachinePoolReconciler) SetupWithManager(ctx context.Context, mgr ctrl.Manager, options controller.Options) error {
	return ctrl.NewControllerManagedBy(mgr).
		WithOptions(options).
		For(&expinfrav1.AWSMachinePool{}).
		Watches(
			&source.Kind{Type: &expclusterv1.MachinePool{}},
			handler.EnqueueRequestsFromMapFunc(machinePoolToInfrastructureMapFunc(expinfrav1.GroupVersion.WithKind("AWSMachinePool"))),
		).
		WithEventFilter(predicates.ResourceNotPausedAndHasFilterLabel(logger.FromContext(ctx).GetLogger(), r.WatchFilterValue)).
		Complete(r)
}

Does the Watches mean it reacts to changes in MachinePool.{spec,status} even if no AWSMachinePool object changed? If yes, that would be great and my proposal could be changed to write the checksums into there as well.

[CecileRobertMichon]

Does the Watches mean it reacts to changes in MachinePool.{spec,status} even if no AWSMachinePool object changed

yes, that's correct