✨ Support Node Auto Placement and Node AF/AAF #3655

zhanggbj · 2025-10-28T07:39:18Z

What this PR does / why we need it:
Support Node Auto Placement and Node AF/AAF

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

- Bump VMOP including Node AF/AAF support - Add NodeAutoPlacement Feature Gate (cherry picked from commit 700c8ae)

(cherry picked from commit cfeb862)

Removes the extra cases for VMG creation, such that VMG is created for: 1. Multiple zones, multiple MDs with no failureDomain 2. Multiple zones, multiple MDs with failureDomain 3. Single zone, existing cluster with no failureDomain MDs Signed-off-by: Sagar Muchhal <[email protected]>

- Updates VMOP API dependency Misc VMG fixes - Use namingStrategy to calculate VM names - Use MachineDeployment names for VMG placement label - Includes all machinedeployments to generate node-pool -> zone mapping Fixes VMG webhook validation error - Adds cluster-name label to Af/AAF spec - re-adds zone topology key back to anti-aff spec Signed-off-by: Sagar Muchhal <[email protected]>

Signed-off-by: Sagar Muchhal <[email protected]>

…gs#71) * Refine VMG controller when generate per-MD zone labels - Skip legacy already-placed VM which do not have placement info - Skip VM which do not have zone info * Apply suggestions from code review --------- Co-authored-by: Sagar Muchhal <[email protected]>

- Sync VSphereMachines during day-2 operations in VMG controller - Only wait for all intended VSphereMachines during initial Cluster creation - Use annotations in VMG for per-md-zone info Signed-off-by: Gong Zhang <[email protected]>

- Add VMG recociler unit test - Bump VMOP due to API change - Filter out VSphereMachine event except create/delete events Signed-off-by: Gong Zhang <[email protected]>

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj · 2025-10-31T06:48:24Z

/test ?

k8s-ci-robot · 2025-10-31T06:48:28Z

@zhanggbj: The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-ci-latest-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-main

/test pull-cluster-api-provider-vsphere-e2e-govmomi-upgrade-1-34-1-35-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-ci-latest-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-main

/test pull-cluster-api-provider-vsphere-e2e-supervisor-upgrade-1-34-1-35-main

/test pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main

/test pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main

/test pull-cluster-api-provider-vsphere-test-main

/test pull-cluster-api-provider-vsphere-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-vsphere-apidiff-main

/test pull-cluster-api-provider-vsphere-janitor-main

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-vsphere-apidiff-main

pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main

pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main

pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main

pull-cluster-api-provider-vsphere-test-main

pull-cluster-api-provider-vsphere-verify-main

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zhanggbj · 2025-10-31T06:49:36Z

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-ci-latest-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-conformance-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-govmomi-upgrade-1-34-1-35-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-ci-latest-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-conformance-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-upgrade-1-34-1-35-main
/test pull-cluster-api-provider-vsphere-e2e-vcsim-govmomi-main
/test pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main
/test pull-cluster-api-provider-vsphere-test-main
/test pull-cluster-api-provider-vsphere-verify-main
/test pull-cluster-api-provider-vsphere-apidiff-main
/test pull-cluster-api-provider-vsphere-janitor-main

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj

All comments are addressed, including the ones missed by mistake last time.

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj · 2025-11-24T03:14:54Z

/test pull-cluster-api-provider-vsphere-e2e-vcsim-supervisor-main

zhanggbj · 2025-11-24T15:00:36Z

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main
/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

zhanggbj · 2025-11-24T15:58:09Z

Hey @fabriziopandini @sbueringer ,
All comments are addressed, including the ones missed by mistake last time. Now CI are green. Please help to take another look, really appreciate!

For the concern about race conditions, I have implemented checks to mitigate race conditions during initial and post-placement, and also have unit teststo cover the scenarios.

zhanggbj · 2025-11-25T09:17:55Z

Pushed a small change and checking UT failures.

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj · 2025-11-25T11:03:14Z

/test pull-cluster-api-provider-vsphere-e2e-supervisor-blocking-main

zhanggbj · 2025-11-26T07:44:01Z

/test pull-cluster-api-provider-vsphere-e2e-govmomi-blocking-main

Since Encryption Class requires API in this newer version, bump vm-operator package Signed-off-by: Gong Zhang <[email protected]>

Signed-off-by: Gong Zhang <[email protected]>

fabriziopandini · 2025-12-11T16:49:31Z

controllers/vmware/virtualmachinegroup_reconciler.go

+				continue
+			}
+			if member.Placement != nil && member.Placement.Zone != "" {
+				log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domanin %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))


nit

Suggested change

log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domanin %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))

log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domain %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))

fabriziopandini · 2025-12-11T17:07:57Z

controllers/vmware/virtualmachinegroup_reconciler_test.go

+		want            bool
+	}{
+		{
+			name: "Should create a VMG if all the expected VSphereMachines exists",


Let's add a unit test ensuring that a deleting vSphereMachine blocks.
I would suggest to use this test case as a setup and add an additional deleting machine

fabriziopandini · 2025-12-11T17:15:13Z

controllers/vmware/virtualmachinegroup_reconciler_test.go

+						clusterv1.ClusterNameLabel: cluster.Name,
+					},
+					Annotations: map[string]string{
+						fmt.Sprintf("%s/%s", vmoperator.ZoneAnnotationPrefix, "md3"): "zone1", // failureDomain for md3 is explicitly set by the user


Let's add an annotation without the ZoneAnnotationPrefix and check it is preserved

fabriziopandini · 2025-12-11T17:15:51Z

controllers/vmware/virtualmachinegroup_reconciler_test.go

+					Namespace: cluster.Namespace,
+					Name:      cluster.Name,
+					UID:       types.UID("uid"),
+					Labels: map[string]string{


Let's add an label different than the ClusterNameLabel and check it is preserved

fabriziopandini · 2025-12-11T17:48:33Z

pkg/services/vmoperator/vmopmachine.go

+				Reason:  infrav1.VSphereMachineVirtualMachineWaitingForVirtualMachineGroupV1Beta2Reason,
+				Message: fmt.Sprintf("Waiting for VSphereMachine's VirtualMachineGroup %s to exist", key),
+			})
+			log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))


Suggested change

log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))

log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s to exist, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))

fabriziopandini · 2025-12-11T17:53:43Z

pkg/services/vmoperator/vmopmachine.go

+		}
+
+		// Set the zone label using the annotation of the per-md zone mapping from VirtualMachineGroup.
+		// This is for new VMs created during day-2 operations when Node Auto Placement is enabled.


Suggested change

// This is for new VMs created during day-2 operations when Node Auto Placement is enabled.

// This is for new VMs created after initial placement decision/with a failureDomain defined by the user.

fabriziopandini · 2025-12-11T17:55:32Z

pkg/services/vmoperator/vmopmachine.go

+			client.MatchingLabels{clusterv1.ClusterNameLabel: supervisorMachineCtx.Cluster.Name}); err != nil {
+			return false, err
+		}
+		othermMDNames := []string{}


nit

Suggested change

othermMDNames := []string{}

otherMDNames := []string{}

fabriziopandini · 2025-12-11T19:00:40Z

pkg/services/vmoperator/vmopmachine_test.go

+			})
+
+			// control plane machine is the machine with the control plane label set
+			Specify("Reconcile valid control plane Machine", func() {


Let's add a check that the CP machine is placed in the expected zone

fabriziopandini · 2025-12-11T19:02:03Z

pkg/services/vmoperator/vmopmachine_test.go

+					vmGroup               *vmoprv1.VirtualMachineGroup
+				)
+
+				BeforeEach(func() {


Looks like we are missing a test for VMG not created yet

fabriziopandini · 2025-12-11T19:10:18Z

pkg/services/vmoperator/vmopmachine_test.go

+
+					machineNotMember.Spec.Bootstrap.DataSecretName = &secretName
+
+					By("VirtualMachine is not created")


Let's check the VM actually does not exist, not only result

zhanggbj and others added 11 commits October 22, 2025 14:19

Bump VMOP and add NodeAutoPlacement Feature Gate

fa60556

- Bump VMOP including Node AF/AAF support - Add NodeAutoPlacement Feature Gate (cherry picked from commit 700c8ae)

Add VMG controller of node auto placement

b52056f

(cherry picked from commit cfeb862)

Removes cluster-name label from affinity rules

9f261d5

Signed-off-by: Sagar Muchhal <[email protected]>

Selectively add node-pool AAF constraint

22d7db0

Signed-off-by: Sagar Muchhal <[email protected]>

Sync VSphereMachines in VMG controller

833cdb5

- Sync VSphereMachines during day-2 operations in VMG controller - Only wait for all intended VSphereMachines during initial Cluster creation - Use annotations in VMG for per-md-zone info Signed-off-by: Gong Zhang <[email protected]>

Add VMG reconciler unit test and bump VMOP

36c3550

- Add VMG recociler unit test - Bump VMOP due to API change - Filter out VSphereMachine event except create/delete events Signed-off-by: Gong Zhang <[email protected]>

Surface error when VM is waiting for VMG

189f6d2

Signed-off-by: Gong Zhang <[email protected]>

Update dependency in packaging&test

8cc8e8a

Signed-off-by: Gong Zhang <[email protected]>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 28, 2025

k8s-ci-robot requested review from MaxRink and rvanderp3 October 28, 2025 07:39

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 28, 2025

zhanggbj changed the title ~~[WIP][DoNotReview]Support Node Auto Placement and Node AF/AAF~~ [WIP][DoNotReview]✨ Support Node Auto Placement and Node AF/AAF Oct 28, 2025

zhanggbj force-pushed the node_auto_placement branch 3 times, most recently from 1160ce9 to 1cd61f9 Compare October 28, 2025 10:31

zhanggbj force-pushed the node_auto_placement branch 7 times, most recently from 22604bf to 7c7c1a7 Compare November 3, 2025 08:55

zhanggbj force-pushed the node_auto_placement branch from ea432ed to dc441f6 Compare November 21, 2025 13:29

Fix UT failure

9cc5762

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj commented Nov 23, 2025

View reviewed changes

This comment was marked as spam.

Sign in to view

zhanggbj force-pushed the node_auto_placement branch 2 times, most recently from 8dabe39 to 19ebccb Compare November 23, 2025 13:27

Refine godoc

b981168

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from 19ebccb to b981168 Compare November 23, 2025 14:16

zhanggbj force-pushed the node_auto_placement branch 3 times, most recently from 42a6d73 to 3ed2e93 Compare November 24, 2025 14:46

zhanggbj force-pushed the node_auto_placement branch from 3ed2e93 to 9b1754a Compare November 25, 2025 06:44

Refine VMG member update to avoid race condition

2ac47d4

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from 9b1754a to 2ac47d4 Compare November 25, 2025 09:25

zhanggbj mentioned this pull request Nov 26, 2025

✨ Add supports to Crypto Spec in VMOp VirtualMachine and PVC #3680

Closed

Update vm-operator package to v1.9.1-0.20251003150112-9b458d311c4c

c41abeb

Since Encryption Class requires API in this newer version, bump vm-operator package Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from fd72f09 to c41abeb Compare November 26, 2025 07:49

zhanggbj and others added 5 commits November 27, 2025 11:46

Added reconcile flow cases

516af84

Signed-off-by: Gong Zhang <[email protected]>

POC AAF

ac0ec4b

Review comments / fixes

822f8f0

Final cleanup

db8095d

Finalize VMG controller and UT

08b8304

Signed-off-by: Gong Zhang <[email protected]>

zhanggbj force-pushed the node_auto_placement branch from b8c569e to 08b8304 Compare November 27, 2025 09:23

fabriziopandini reviewed Dec 12, 2025

View reviewed changes

	log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domanin %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))
	log.Info(fmt.Sprintf("MachineDeployment %s has been placed to failure domain %s", md.Name, member.Placement.Zone), "MachineDeployment", klog.KObj(&md))

	log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))
	log.V(4).Info(fmt.Sprintf("Waiting for VirtualMachineGroup %s to exist, requeueing", key.Name), "VirtualMachineGroup", klog.KRef(key.Namespace, key.Name))

	// This is for new VMs created during day-2 operations when Node Auto Placement is enabled.
	// This is for new VMs created after initial placement decision/with a failureDomain defined by the user.


		machineNotMember.Spec.Bootstrap.DataSecretName = &secretName

		By("VirtualMachine is not created")

✨ Support Node Auto Placement and Node AF/AAF #3655

Are you sure you want to change the base?

✨ Support Node Auto Placement and Node AF/AAF #3655

Conversation

zhanggbj commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhanggbj commented Oct 31, 2025

Uh oh!

k8s-ci-robot commented Oct 31, 2025

Uh oh!

zhanggbj commented Oct 31, 2025

Uh oh!

zhanggbj left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

zhanggbj commented Nov 24, 2025

Uh oh!

zhanggbj commented Nov 24, 2025

Uh oh!

zhanggbj commented Nov 24, 2025

Uh oh!

zhanggbj commented Nov 25, 2025

Uh oh!

zhanggbj commented Nov 25, 2025

Uh oh!

zhanggbj commented Nov 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zhanggbj commented Oct 28, 2025 •

edited

Loading