WIP: PoC Custom pool booting

[djoshy]

Disclaimer: I mainly just wrote this for fun, to see if it was easy to do 😄 I might be missing certain use cases, but just wanted to open it up for wider review.

- What I did

• MCS applies a new annotation(machineconfiguration.openshift.io/firstPivotConfig) that stores the very first config that was served to the node.
• MCC checks this annotation, sees if the MC served first to the node in question was for a custom pool, and applies the custom label to prevent it from getting queued for an update back to worker pool. It also applies another annotation(machineconfiguration.openshift.io/customPoolLabelsApplied) for bookkeeping purposes after this, so it doesn't attempt to label the node again if it was removed by some other actor.

- How to verify it

• Create a cluster with this PR.
• Create a custom MCP named infra:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: infra
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/infra: ""

• Make a copy one of your machinesets to disk. Make the following changes to it:
  - Edit its name to be unique.
  - Change the MachineSet's userDataSecret field to infra-user-data-managed if you used the infra MCP I provided in the previous step. If you used another MCP name, use $MCP_NAME-user-data-managed instead.
  - I also changed the cluster-api-machineset labels to match, but I am unsure if it is needed.
• Apply your edited machineset to the cluster. You should a new node scale-up and join the cluster. It will appear to join the worker pool at first, but then get moved to the infra pool. No update/reboot should take place in this transition.
• Observe the MCC logs:

I1016 19:26:21.404751       1 node_controller.go:677] Pool worker[zone=us-east4-c]: node djoshy-dev-101-nwpzl-worker-infra-kthf5: changed annotation machineconfiguration.openshift.io/reason = 
I1016 19:26:40.081200       1 node_controller.go:677] Pool worker[zone=us-east4-c]: node djoshy-dev-101-nwpzl-worker-infra-kthf5: Reporting ready
I1016 19:26:40.131083       1 node_controller.go:677] Pool worker[zone=us-east4-c]: node djoshy-dev-101-nwpzl-worker-infra-kthf5: changed taints
I1016 19:26:41.565210       1 node_controller.go:677] Pool worker[zone=us-east4-c]: node djoshy-dev-101-nwpzl-worker-infra-kthf5: changed taints
I1016 19:26:45.082788       1 node_controller.go:1348] Pool worker: selected candidate node djoshy-dev-101-nwpzl-worker-infra-kthf5
I1016 19:26:45.082843       1 node_controller.go:667] Pool worker: 1 candidate nodes in 1 zones for update, capacity: 1
I1016 19:26:45.082852       1 node_controller.go:1486] Applying node selector labels from custom pool infra to node djoshy-dev-101-nwpzl-worker-infra-kthf5: map[node-role.kubernetes.io/infra:]
I1016 19:26:45.096179       1 node_controller.go:1500] Successfully applied custom pool labels to node djoshy-dev-101-nwpzl-worker-infra-kthf5
I1016 19:26:45.096220       1 node_controller.go:1417] node djoshy-dev-101-nwpzl-worker-infra-kthf5 has been moved to pool infra, dropping from candidate list

• Profit!!! 💰 You have booted a new node directly into your custom pool! 🎉

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[yuqi-zhang]

The general workflow sounds fine to me, definitely aligns with our general discussions around how to make this work.

The only real "functionality" we're adding is to have the node request additional labels via annotations, which should be safe I feel like since if the CSR went through to make it a node, we should already trust it.

I guess technically the workflow today probably sees users adding labels via machine/machineset objects, so at worst we'd duplicate that part? Either way it shouldn't cause an error

[yuqi-zhang]

Oh, for some reason I forgot we did this, neat

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [djoshy,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jlhuilier-1a]

Hello, Just for information we started to use the managed user-data in our machineset definition (To get the correct final config at startup). It works however we rapidly faced issues with clusters that were created with previous version of openshift.
The issue is that today the VM image used is created only at install time with the corresponding unmanaged worker-user-data and master-user-data. Managed user-data on the other hand are evolving with OCP version and you end-up having ignition version unknown to the VM image (Hence machine stays as starting).

[djoshy]

Hi @jlhuilier-1a ! You're correct, the reason you're running into that is because your boot images are likely out of date. The *-user-data stub needs to be compatible with the boot image referenced by your machineset. Some more context can be found here. The boot image update mechanism described in this document also attempts to upgrade these secrets to the newest version for the same reason.

As for this PR, the instructions described were just the easiest path to test 😄 You could also copy an existing user-data secret (say, call it infra-user-data) and then edit the MCS endpoint within the stub to target the infra pool. Then, edit the machineset to reference that instead; this should work in a similar fashion. When we do eventually bring this feature to GA, we will be sure to make a note of this in the workflow - thanks for bringing this up, appreciate your review!