✨ Support Node Auto Placement and Node AF/AAF #3655

zhanggbj · 2025-10-28T07:39:18Z

What this PR does / why we need it:
Support Node Auto Placement and Node AF/AAF

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

- Bump VMOP including Node AF/AAF support - Add NodeAutoPlacement Feature Gate (cherry picked from commit 700c8ae)

(cherry picked from commit cfeb862)

Removes the extra cases for VMG creation, such that VMG is created for: 1. Multiple zones, multiple MDs with no failureDomain 2. Multiple zones, multiple MDs with failureDomain 3. Single zone, existing cluster with no failureDomain MDs Signed-off-by: Sagar Muchhal <[email protected]>

- Updates VMOP API dependency Misc VMG fixes - Use namingStrategy to calculate VM names - Use MachineDeployment names for VMG placement label - Includes all machinedeployments to generate node-pool -> zone mapping Fixes VMG webhook validation error - Adds cluster-name label to Af/AAF spec - re-adds zone topology key back to anti-aff spec Signed-off-by: Sagar Muchhal <[email protected]>

Signed-off-by: Sagar Muchhal <[email protected]>

…gs#71) * Refine VMG controller when generate per-MD zone labels - Skip legacy already-placed VM which do not have placement info - Skip VM which do not have zone info * Apply suggestions from code review --------- Co-authored-by: Sagar Muchhal <[email protected]>

- Sync VSphereMachines during day-2 operations in VMG controller - Only wait for all intended VSphereMachines during initial Cluster creation - Use annotations in VMG for per-md-zone info Signed-off-by: Gong Zhang <[email protected]>

- Add VMG recociler unit test - Bump VMOP due to API change - Filter out VSphereMachine event except create/delete events Signed-off-by: Gong Zhang <[email protected]>

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj · 2025-10-31T06:48:24Z

/test ?

k8s-ci-robot · 2025-10-31T06:48:28Z

@zhanggbj: The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-ci-latest-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-upgrade-1-34-1-35-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-ci-latest-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-upgrade-1-34-1-35-main

/test pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main

/test pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main

/test pull-cluster-api-provider-vsphere-test-main

/test pull-cluster-api-provider-vsphere-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-vsphere-apidiff-main

/test pull-cluster-api-provider-vsphere-janitor-main

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-vsphere-apidiff-main

pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main

pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main

pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main

pull-cluster-api-provider-vsphere-test-main

pull-cluster-api-provider-vsphere-verify-main

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zhanggbj · 2025-10-31T06:49:36Z

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-ci-latest-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-upgrade-1-34-1-35-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-ci-latest-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-upgrade-1-34-1-35-main
/test pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main
/test pull-cluster-api-provider-vsphere-test-main
/test pull-cluster-api-provider-vsphere-verify-main
/test pull-cluster-api-provider-vsphere-apidiff-main
/test pull-cluster-api-provider-vsphere-janitor-main

controllers/vmware/virtualmachinegroup_controller.go

k8s-ci-robot · 2025-11-12T08:52:42Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from fabriziopandini. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

controllers/vmware/virtualmachinegroup_reconciler.go

- Update to watch Cluster as primary - Decouple functions - Update to accurate match when checking if VM is a member of VMG - Update UT - Refine godoc - Miscellaneous Signed-off-by: Gong Zhang <[email protected]>

controllers/vmware/virtualmachinegroup_controller.go

DanielXiao

A question which may be picky, you don't have to solve it.

sbueringer · 2025-11-13T08:58:19Z

config/rbac/role.yaml

+- apiGroups:
+  - vmoperator.vmware.com
+  resources:
+  - virtualmachinegroups/status


Let's just add all verbs on the marker for virtualmachineservices/status so it will be grouped in the RBAC rule above

(not sure how much sense this permission even makes, nobody does a get status :), but let's align to what we do for the other vmop resources)

Updated virtualmachineservices/status with all markers so it could be grouped.

Question: Seems CAPV has permission it doesn't truly use. Should we keep the permission minimal just as what controller used?

Feel free to open a follow-up issue

Filed an issue for tracking: #3681

sbueringer · 2025-11-13T09:01:49Z

main.go

 		return err
 	}

+	if feature.Gates.Enabled(feature.NamespaceScopedZones) && feature.Gates.Enabled(feature.NodeAutoPlacement) {


I'm okay with not adding additional validation here

(I think the validation here would mean we would need an additional validating webhook on the Cluster object)

sbueringer · 2025-11-13T09:13:25Z

controllers/vmware/virtualmachinegroup_controller.go

+		Named("virtualmachinegroup").
+		Watches(
+			&vmoprv1.VirtualMachineGroup{},
+			handler.EnqueueRequestForOwner(mgr.GetScheme(), reconciler.Client.RESTMapper(), &clusterv1.Cluster{}),


Suggested change

handler.EnqueueRequestForOwner(mgr.GetScheme(), reconciler.Client.RESTMapper(), &clusterv1.Cluster{}),

handler.EnqueueRequestForOwner(mgr.GetScheme(), reconciler.Client.RESTMapper(), &clusterv1.Cluster{}),

ctrlbldr.WithPredicates(predicates.ResourceIsChanged(mgr.GetScheme(), predicateLog)),

We have not been using this in CAPV up until now, but let's use it here to already improve this new reconciler (I have a task to follow-up for others soon'ish).

Per default the reconciler will get resyncs from all informers. i.e. every 10m everything gets reconciled because of the Cluster resync and additionally every 10m everything gets reconciled because of the VirtualMachineGroup resync.

In core CAPI we drop events from resyncs of secondary watches to avoid this issue

(it's not needed for VSphereMachine below as we already drop all updates)

Good catch. This help to reduce the resync noices.

sbueringer · 2025-11-13T09:19:01Z

controllers/vmware/virtualmachinegroup_controller.go

+	vmg := &vmoprv1.VirtualMachineGroup{}
+	err := r.Client.Get(ctx, apitypes.NamespacedName{Namespace: vSphereMachine.Namespace, Name: clusterName}, vmg)
+
+	if err != nil {
+		return nil
+	}


I think we should not have this kind of logic in a mapper.

Let's deal with this only in the controller.

It's much more obvious there and the cache here can be out of sync.

Let's then use vSphereMachine.Namespace and clusterName for the request below

(please also update the godoc of this func)

Make senses. Fixed it and also added predict in watch for VSphereMachine to only handle VSphereMachines which have MachineDeployment label.

sbueringer · 2025-11-13T09:22:27Z

controllers/vmware/virtualmachinegroup_reconciler.go

+	// reconciliationDelay is the delay time for requeueAfter.
+	reconciliationDelay = 10 * time.Second
+	// ZoneAnnotationPrefix is the prefix used for placement decision annotations which will be set on VirtualMachineGroup.
+	ZoneAnnotationPrefix = "zone.cluster.x-k8s.io"


I think we have to namespace this to CAPV as it's not a Cluster API wide standard annotation

(TBD with @fabriziopandini what exactly we should use instead)

Discussed, let's use zone.vmware.infrastructure.cluster.x-k8s.io

pkg/services/vmoperator/vmopmachine.go

fabriziopandini

Partial review only.

A few themes are emerging and it will be nice if they are addressed across the changeset:

we should add godoc keeping track of why we are doing things in a specific way, because this will help future us when doing maintenance
we should use consistently machineDeployment instead of nodePool, failureDomain instead of zone (I know there is tech debt, but we don't have to add on top, possibly improve)
logs should align to guidelines

pkg/services/vmoperator/vmopmachine.go

fabriziopandini · 2025-11-13T13:05:48Z

pkg/services/vmoperator/vmopmachine.go

 			vmOperatorVM = typedModified
 		}

+		if affinityInfo != nil && affinityInfo.affinitySpec != nil {


Let's try to document why we are doing stuff, because this is the background that will be most valuable for future maintenance, eg.

Suggested change

if affinityInfo != nil && affinityInfo.affinitySpec != nil {

// Set Affinity and Group info on a VM only on creation,

// because, changing those info after placement won't have any effect.

if affinityInfo != nil && affinityInfo.affinitySpec != nil {

I would also probably revert the if, to check "if not set" (set on creation) first

Added comments for clarification that affinity only impact VM locations during day-1.

I would also probably revert the if, to check "if not set" (set on creation) first

Could you help clarify this one?

I tried to improve your comment to prevent confusion between the comment taking about when the anti affinity rules have effect and the code always setting those rules

pkg/services/vmoperator/vmopmachine.go

fabriziopandini · 2025-11-13T13:18:55Z

pkg/services/vmoperator/vmopmachine.go

+type affinityInfo struct {
+	affinitySpec  *vmoprv1.AffinitySpec
+	vmGroupName   string
+	failureDomain *string


Are we should we need a pointer here? (If i'm not wrong there is no difference between this being nil or empty)

I think using failureDomain string works here.
CAPV is using failureDomain * stringhttps://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/apis/v1beta1/vspheremachine_types.go#L117 but CAPI is using failureDomain string, how about we keep it consistent in CAPV? but I don't have strong opinion on this.

Let use string here, it is a private struct and there are no direct relations with the API types (also API types cannot be changed, it is a breaking change)

fabriziopandini · 2025-11-13T13:21:09Z

pkg/services/vmoperator/vmopmachine.go


+// affinityInfo is an internal to store VM affinity information.
+type affinityInfo struct {
+	affinitySpec  *vmoprv1.AffinitySpec


Are we sure we need a pointer here? looking at usage, whenever affinityInfo is set, also affinitySpec is set

Oh right, whenever we set affinityInfo, we'll always set affinitySpec. Seems only checking affinityInfo != nil is fine.

However, since AffinitySpec contains lots of rules and arrays of MachineDeployments. Is it better to keep it *vmoprv1.AffinitySpec for less Mem usage?

I would prefer readability to micro optimization (the struct is used only locally)

pkg/services/vmoperator/vmopmachine.go

controllers/vmware/virtualmachinegroup_reconciler.go

fabriziopandini · 2025-11-13T20:52:10Z

controllers/vmware/virtualmachinegroup_reconciler.go

+
+		// Calculate expected Machines of all MachineDeployments.
+		// CAPV retrieves placement decisions from the VirtualMachineGroup to guide
+		// day-2 VM placement. At least one VM is required for each MachineDeployment.


At least one VM is required for each MachineDeployment.

I'm wondering if this assumption holds when using autoscaler from zero... 🤔

cc @sbueringer

Since this check only happens when VMG is not created yet, so for day-2 scale from zero, it won't go into this case.
In addition for day-1, we'll block create Cluster with MD.replica == 0.

For day-2, since we'll get vmg successfully in line , so we won't run into this line.

does it means that a user can't create a cluster with automatic node placement and a MachineDeployment configured with autoscaler to zero enabled?
If this is the case, please add a comment

controllers/vmware/virtualmachinegroup_reconciler.go

fabriziopandini · 2025-11-13T21:02:50Z

controllers/vmware/virtualmachinegroup_reconciler.go

+		// Wait for all intended VSphereMachines corresponding to MachineDeployment to exist only during initial Cluster creation.
+		current := int32(len(current))
+		if current < expected {
+			log.Info("current VSphereMachines do not match expected", "Expected:", expected,


Should this message start with Waiting for...

pkg/services/vmoperator/vmopmachine.go

controllers/vmware/virtualmachinegroup_reconciler.go

zhanggbj · 2025-11-17T09:07:34Z

@sbueringer @fabriziopandini
I've addressed above comments, please help to take another review, thanks!

BTW, I'm still improving logging and godoc incrementally, feel free to point out if anything missing.

fabriziopandini · 2025-11-14T09:30:01Z

controllers/vmware/virtualmachinegroup_reconciler.go

+		Name:      cluster.Name,
+	}
+
+	if err := r.Client.Get(ctx, *key, desiredVMG); err != nil {


Suggested change

if err := r.Client.Get(ctx, *key, desiredVMG); err != nil {

// If the VM group does not exist yet

if err := r.Client.Get(ctx, *key, desiredVMG); err != nil {

Added comments.

fabriziopandini · 2025-11-18T13:28:09Z

pkg/services/vmoperator/vmopmachine.go

+		}
+		mdNames := []string{}
+		for _, machineDeployment := range machineDeployments.Items {
+			if machineDeployment.Spec.Template.Spec.FailureDomain == "" && machineDeployment.Name != mdName {


is it correct to not consider machine deployments without FailureDomain?

As a user, I would like that anti affinity considers all the MachineDeployments, not only a subset (only exception should be the MachineDeployment the machine belongs to)

cc @sbueringer

This is because with current VMOP design, MachineDeployment with failureDomain specified will skip placement process. So I think this is fine.

fabriziopandini · 2025-11-18T13:28:27Z

pkg/services/vmoperator/vmopmachine.go

+			client.MatchingLabels{clusterv1.ClusterNameLabel: supervisorMachineCtx.Cluster.Name}); err != nil {
+			return false, err
+		}
+		mdNames := []string{}


may be

Suggested change

mdNames := []string{}

otherMDNames := []string{}

fabriziopandini · 2025-11-18T13:38:16Z

pkg/services/vmoperator/vmopmachine.go

+		// Set the zone label using the annotation of the per-md zone mapping from VMG.
+		// This is for new VMs created during day-2 operations in VC 9.1.


This comment seems not yet addressed

fabriziopandini · 2025-11-18T13:38:38Z

pkg/services/vmoperator/vmopmachine.go

+			return true, nil
+		}
+
+		affInfo = affinityInfo{


This comment seems not yet addressed

fabriziopandini · 2025-11-18T15:01:01Z

controllers/vmware/virtualmachinegroup_reconciler.go

+	}
+
+	// If ControlPlane haven't initialized, requeue it since CAPV will only start to reconcile VSphereMachines of
+	// MachineDeployment until ControlPlane is initialized.


may be

Suggested change

// MachineDeployment until ControlPlane is initialized.

// MachineDeployment after ControlPlane is initialized.

fabriziopandini · 2025-11-18T15:02:58Z

controllers/vmware/virtualmachinegroup_reconciler.go

+
+		// Wait for all intended VSphereMachines corresponding to MachineDeployment to exist only during initial Cluster creation.
+		current := int32(len(current))
+		if current < expected {


This comment seems not yet addressed

fabriziopandini · 2025-11-18T15:06:49Z

controllers/vmware/virtualmachinegroup_reconciler.go

+	// Set the BootOrder spec as the
+	desiredVMG.Spec.BootOrder = []vmoprv1.VirtualMachineGroupBootOrderGroup{
+		{
+			Members: members,


This comment seems not yet addressed

fabriziopandini · 2025-11-18T15:11:05Z

controllers/vmware/virtualmachinegroup_reconciler.go

+
+		// Check if this VM belongs to any of our target MachineDeployments.
+		// Annotation format is "zone.cluster.x-k8s.io/{machine-deployment-name}".
+		for _, md := range machineDeployments {


This comment seems not yet addressed

fabriziopandini · 2025-11-18T15:11:58Z

controllers/vmware/virtualmachinegroup_reconciler.go

+
+// GenerateVirtualMachineName generates the name of a VirtualMachine based on the naming strategy.
+// Duplicated this logic from pkg/services/vmoperator/vmopmachine.go.
+func GenerateVirtualMachineName(machineName string, namingStrategy *vmwarev1.VirtualMachineNamingStrategy) (string, error) {


This comment seems not yet addressed

- Refine VMG controller watch - Handle race conditions in VMG controller by gating member update - Refine data struct for VM Affinity config - Refine UT - Refine naming, logging, godoc - Miscellaneous Signed-off-by: Gong Zhang <[email protected]>

k8s-ci-robot · 2025-11-21T13:41:41Z

@zhanggbj: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-vsphere-test-main	`dc441f6`	link	true	`/test pull-cluster-api-provider-vsphere-test-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

zhanggbj and others added 11 commits October 22, 2025 14:19

Bump VMOP and add NodeAutoPlacement Feature Gate

fa60556

- Bump VMOP including Node AF/AAF support - Add NodeAutoPlacement Feature Gate (cherry picked from commit 700c8ae)

Add VMG controller of node auto placement

b52056f

(cherry picked from commit cfeb862)

Removes cluster-name label from affinity rules

9f261d5

Signed-off-by: Sagar Muchhal <[email protected]>

Selectively add node-pool AAF constraint

22d7db0

Signed-off-by: Sagar Muchhal <[email protected]>

Sync VSphereMachines in VMG controller

833cdb5

- Sync VSphereMachines during day-2 operations in VMG controller - Only wait for all intended VSphereMachines during initial Cluster creation - Use annotations in VMG for per-md-zone info Signed-off-by: Gong Zhang <[email protected]>

Add VMG reconciler unit test and bump VMOP

36c3550

- Add VMG recociler unit test - Bump VMOP due to API change - Filter out VSphereMachine event except create/delete events Signed-off-by: Gong Zhang <[email protected]>

Surface error when VM is waiting for VMG

189f6d2

Signed-off-by: Gong Zhang <[email protected]>

Update dependency in packaging&test

8cc8e8a

Signed-off-by: Gong Zhang <[email protected]>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 28, 2025

k8s-ci-robot requested review from MaxRink and rvanderp3 October 28, 2025 07:39

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 28, 2025

zhanggbj changed the title ~~[WIP][DoNotReview]Support Node Auto Placement and Node AF/AAF~~ [WIP][DoNotReview]✨ Support Node Auto Placement and Node AF/AAF Oct 28, 2025

zhanggbj force-pushed the node_auto_placement branch 3 times, most recently from 1160ce9 to 1cd61f9 Compare October 28, 2025 10:31

zhanggbj force-pushed the node_auto_placement branch 7 times, most recently from 22604bf to 7c7c1a7 Compare November 3, 2025 08:55

DanielXiao reviewed Nov 12, 2025

View reviewed changes

controllers/vmware/virtualmachinegroup_controller.go Outdated Show resolved Hide resolved

DanielXiao reviewed Nov 12, 2025

View reviewed changes

controllers/vmware/virtualmachinegroup_controller.go Outdated Show resolved Hide resolved

zhanggbj force-pushed the node_auto_placement branch 8 times, most recently from afd77d2 to 233cf64 Compare November 12, 2025 16:13

DanielXiao reviewed Nov 13, 2025

View reviewed changes

controllers/vmware/virtualmachinegroup_reconciler.go Outdated Show resolved Hide resolved

Refine VMG controller

e3078c9

- Update to watch Cluster as primary - Decouple functions - Update to accurate match when checking if VM is a member of VMG - Update UT - Refine godoc - Miscellaneous Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from 233cf64 to e3078c9 Compare November 13, 2025 03:35

DanielXiao reviewed Nov 13, 2025

View reviewed changes

controllers/vmware/virtualmachinegroup_controller.go Outdated Show resolved Hide resolved

DanielXiao reviewed Nov 13, 2025

View reviewed changes

sbueringer reviewed Nov 13, 2025

View reviewed changes

fabriziopandini reviewed Nov 13, 2025

View reviewed changes

zhanggbj mentioned this pull request Nov 14, 2025

Eliminate requeueAfter Usage for Scalability #3672

Open

1 task

zhanggbj force-pushed the node_auto_placement branch from bab7415 to 1524d87 Compare November 17, 2025 08:00

fabriziopandini reviewed Nov 18, 2025

View reviewed changes

zhanggbj force-pushed the node_auto_placement branch 4 times, most recently from 3b5ec17 to ea432ed Compare November 21, 2025 09:55

Address review comments

dc441f6

- Refine VMG controller watch - Handle race conditions in VMG controller by gating member update - Refine data struct for VM Affinity config - Refine UT - Refine naming, logging, godoc - Miscellaneous Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from ea432ed to dc441f6 Compare November 21, 2025 13:29

	handler.EnqueueRequestForOwner(mgr.GetScheme(), reconciler.Client.RESTMapper(), &clusterv1.Cluster{}),
	handler.EnqueueRequestForOwner(mgr.GetScheme(), reconciler.Client.RESTMapper(), &clusterv1.Cluster{}),
	ctrlbldr.WithPredicates(predicates.ResourceIsChanged(mgr.GetScheme(), predicateLog)),

	if err := r.Client.Get(ctx, *key, desiredVMG); err != nil {
	// If the VM group does not exist yet
	if err := r.Client.Get(ctx, *key, desiredVMG); err != nil {

		// Set the zone label using the annotation of the per-md zone mapping from VMG.
		// This is for new VMs created during day-2 operations in VC 9.1.

	// MachineDeployment until ControlPlane is initialized.
	// MachineDeployment after ControlPlane is initialized.

✨ Support Node Auto Placement and Node AF/AAF #3655

Are you sure you want to change the base?

✨ Support Node Auto Placement and Node AF/AAF #3655

Conversation

zhanggbj commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhanggbj commented Oct 31, 2025

Uh oh!

k8s-ci-robot commented Oct 31, 2025

Uh oh!

zhanggbj commented Oct 31, 2025

Uh oh!

Uh oh!

Uh oh!

k8s-ci-robot commented Nov 12, 2025

Uh oh!

Uh oh!

Uh oh!

DanielXiao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fabriziopandini left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhanggbj Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhanggbj commented Oct 28, 2025 •

edited

Loading

fabriziopandini left a comment •

edited

Loading

zhanggbj Nov 14, 2025 •

edited

Loading