WIP node setup by bcho · Pull Request #77 · Azure/AKSFlexNode

bcho · 2026-02-12T07:31:04Z

No description provided.

jim-minter · 2026-02-12T15:37:10Z

docs/node-env.md

+- We will limit the support scope to Linux-based nodes and focus on Ubuntu distro for now.
+  This is because Ubuntu is the widely and commonly available Linux distribution
+  across the target environments.


I think we should consider focusing on Ubuntu and Azure Linux from the start.

for 3p clouds do we want to offer AzLinux?

jim-minter · 2026-02-12T15:38:00Z

docs/node-env.md

+  * `containerd` w/ 2.0+ version;
+  * `runc`


Both? Or one?

jim-minter · 2026-02-12T15:39:37Z

docs/node-env.md

+  * TLS bootstrap configurations;
+  * Other cloud provider binaries;
+- NFTables / IPtables installed for Kubernetes network policies;
+- Network forward, IP masquerade and bridge settings configured for Kubernetes networking;


VPN components?

yeah I think we need to check what kind of VPN components are needed here. But for now I don't have a concrete answer so I left it out

jim-minter · 2026-02-12T15:43:09Z

docs/node-env.md

+- Detailed GPU device plugin requirements and enablement strategies will be addressed in
+  a separate document.
+
+## Baseline Environment Requirements


Maybe it would be possible to more clearly separate (a) binaries and configuration, and (b) stuff that could/should be baked in (if we're baking) and stuff that might not be?

jim-minter · 2026-02-12T15:47:22Z

docs/node-env.md

+
+**Expected behaviors**:
+
+- Produced image is **immutable** and **reproducible** giving the same inputs.


Immutable? Well, it's an image, and we don't prevent files being modified at runtime?
Reproducible? As far as the contents of the filesystem, maybe -- byte for byte at an image level is probably hard.

immutable means we don't update the same published version after that. Reproducible means we can rebuild the same image with same set of components / binaries in any time. The other files inside the image that are not critical to the kubelet functionalitity, then it's fine to be drifted.

will find a way to document this part in the doc. I called them out because I have seen a few incidents in agent baker side caused by changed of the inputs (artifact naming change from package registry for example) caused outage. If we can find a good contract to limit and pin the critical components, then incidents like that could be avoided. cc @cameronmeissner

added two footnotes to explain the terms here

jim-minter · 2026-02-12T15:48:07Z

docs/node-env.md

+
+**Inputs**:
+
+- Cluster endpoint (API server URL, CA bundle)


Perhaps network/VPN configuration/credentials?

Perhaps identity credentials

added two more items to cover them

jim-minter · 2026-02-12T15:49:25Z

docs/node-env.md

+
+- All failure handling mechanisms from both Node VHD Image Baking and Node Bootstrapping
+
+### Node Rebooting & Repairing


Is any of the stuff below really in scope for this document?

I added these operations to make sure the tool we are building here will support for all these scenarios.

jim-minter · 2026-02-12T15:50:22Z

docs/node-env.md

+  * `containerd` w/ 2.0+ version;
+  * `runc`
+- Kubernetes components:
+  * `kubelet` matching with the target worker node version;


Will images be Kubernetes-version specific, or will one image support multiple k8s versions?

This is actually a debatable question, in current agent baker, we bake multiple k8s versions as AKS has multiple versions support. But I think for flex node, we can do in a more limited way so we can reduce the support matrix while providing more stable functionality.

cameronmeissner · 2026-02-12T16:53:02Z

docs/node-env.md

+- Kubernetes components:
+  * `kubelet` matching with the target worker node version;
+  * Control plane public CA certificate(s);
+  * TLS bootstrap configurations;


for flex nodes, it looks like we plan on using bootstrap tokens (at least as a start) - have we thought about mechanisms by which we could avoid bootstrap tokens that would work across all clouds / on-prem environments? (this is probably a project in and of itself, though just curious)

not every cloud support the same set of features, hence I just put "TLS bootstrap configurations" here. It can be static token or Arc or something similar to secure TLS bootstrap setup.

cameronmeissner · 2026-02-12T16:53:46Z

docs/node-env.md

+
+### Additional Requirements
+
+- Node identity for identifying and authenticating the node to cluster control plane;


node identity - are you referring to the client certificate obtained through TLS bootstrapping here?

cameronmeissner · 2026-02-12T16:57:13Z

docs/node-env.md

+
+### Node VHD Image Baking
+
+**Purpose**: Produce a base node image (VHD or similar) that satisfies baseline


have we thought about how distribution of said base node image would look like?

yeah unfortunately, different clouds will have different ways for doing this. I guess we will end up with maintaining a couple of supported node images in every cloud

cameronmeissner · 2026-02-12T16:59:32Z

docs/node-env.md

+
+### Node Bootstrapping w/ Baking
+
+**Purpose**: In environments without pre-baked images, the bootstrapping process


Artur and I have talked about embedding the provisioning scripts we have in AgentBaker directly into aks-node-controller to accomplish this - I haven't looked around the rest of this repo yet, but it seems like you're dealing with component installation all natively in Golang from the start? If so you wouldn't need to do something like script embedding, though just thought that would be worth mentioning

I think we will end up with having a binary embeded to the node for doing things similar to the aks-node-controller

bcho added 6 commits February 11, 2026 15:59

feat: add qemu vm helper

c8fc59d

chore: simlify signal handling

b9860b2

fix: RuntimeConfig -> RuncConfig

232a107

fix: add minimal bootstrap process

4c1b2f6

doc: describe agent node host environment baseline

530e677

doc: refine wordings

17991a8

bcho temporarily deployed to e2e-testing February 12, 2026 07:31 — with GitHub Actions Inactive

doc: wording fix

3e0cb10

bcho temporarily deployed to e2e-testing February 12, 2026 07:47 — with GitHub Actions Inactive

bcho changed the title ~~WIP node setpu~~ WIP node setup Feb 12, 2026

jim-minter reviewed Feb 12, 2026

View reviewed changes

cameronmeissner reviewed Feb 12, 2026

View reviewed changes

doc: feedbacks

5fb712b

bcho temporarily deployed to e2e-testing February 12, 2026 18:34 — with GitHub Actions Inactive

Merge remote-tracking branch 'origin/main' into hbc/stretch-setup

efc21d0

bcho temporarily deployed to e2e-testing February 12, 2026 19:40 — with GitHub Actions Inactive

bcho mentioned this pull request Feb 17, 2026

doc: add components api doc #85

Open


		Expected behaviors:

		- Produced image is immutable and reproducible giving the same inputs.


		- All failure handling mechanisms from both Node VHD Image Baking and Node Bootstrapping

		### Node Rebooting & Repairing


		### Additional Requirements

		- Node identity for identifying and authenticating the node to cluster control plane;


		### Node VHD Image Baking

		Purpose: Produce a base node image (VHD or similar) that satisfies baseline


		### Node Bootstrapping w/ Baking

		Purpose: In environments without pre-baked images, the bootstrapping process

Conversation

bcho commented Feb 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants