From 00ae11268ef84e134ea41d7701262d2d246decd4 Mon Sep 17 00:00:00 2001 From: Marc Power Date: Fri, 26 Jun 2026 01:04:31 +0000 Subject: [PATCH] feat: support Windows node provisioning via the AKS Machine API Enable Karpenter (Azure provider) to provision Windows worker nodes when running in the AKS Machine API provision mode (PROVISION_MODE=aksmachineapi / aksmachineapiheaderbatch). The Karpenter controller itself continues to run as a Linux pod; only the provisioned worker nodes are Windows. This is the mode AKS ships as managed Node Auto Provisioning (NAP), and the AKS RP / Node Provisioning Service already handles Windows server-side (OSType/OSSKU, AgentPoolWindowsProfile, and admin-credential sourcing from the cluster's windowsProfile), so no RP-side changes are required. Previously every code path was hard-coded to Linux: instance types advertised kubernetes.io/os=linux unconditionally, there was no Windows image family, and the Machine builder always emitted OSType=Linux. As a result the scheduler never matched a Windows pod to any offering and no Windows node was ever created. What this change does --------------------- Scheduling - Derive the kubernetes.io/os requirement from the NodeClass image family via a new v1beta1.GetOSForImageFamily helper, instead of hard-coding Linux, so a Windows NodeClass advertises os=windows and the scheduler bin-packs Windows pods onto provisionable nodes. OS values are defined as constants (v1beta1.OSLinux / OSWindows) rather than magic strings. NodeClass API (v1beta1) - Add Windows image families: Windows2022 and Windows2025, each mapped 1:1 to its os-sku label value (kubernetes.azure.com/os-sku), plus a WindowsFamilies set and a case-insensitive v1beta1.IsWindowsImageFamily helper used by every family check so casing never silently routes a Windows family down the Linux path. - Extend the imageFamily CEL enum and register the Windows os-sku values in WellKnownValuesForRequirements. - Reject FIPS and linuxOSConfig for Windows families via CEL validation; the AKSNodeClass CRD (and the Helm CRD copy) are regenerated accordingly. Image selection - Add a Windows image family (pkg/providers/imagefamily/windows.go) that returns Windows images from the AKS-managed shared image gallery (AKSWindows) in gen2-then-gen1, amd64 order. Windows is SIG-only (no community gallery) and has no FIPS variants. The bootstrap methods return a clear error because Windows is only supported via the Machine API path, not the in-provider bootstrappers. - Include the AKSWindows gallery in FilteredNodeImages; it was previously filtered to the Ubuntu and Azure Linux galleries only, which silently dropped every Windows image. - Convert Windows node image versions correctly: the AKSWindows image definition prefix ("windows-") is stripped so the result matches the AKS node-image-version form (e.g. AKSWindows-2022-containerd-gen2-). AKS Machine API provisioning - Set OSType=Windows and OSSKU=Windows2022/Windows2025 for Windows NodeClasses, omit the LinuxProfile, and force EnableFIPS=false. - Omit NodeImageVersion for Windows machines: the RP's input parser splits on "-" and expects exactly gallery-name-version, but Windows image names contain hyphens, so an explicit value is rejected. Leaving it empty lets the RP resolve the latest image from the OSSKU. - Generate a short, deterministic, hyphen-free AKS machine name for Windows to satisfy the Windows NetBIOS computer-name limit enforced by the Machine API. The budget is pool-aware, mirroring the AKS RP machine-name validation: <=12 chars in the reserved NAP pool (aksmanagedap) and <=5 chars in a custom / self-hosted machines pool (whose RP-composed VM name also embeds the pool name). The name is split into dedicated GetLinuxAKSMachineName and GetWindowsAKSMachineName(maxLen) helpers chosen at the call site; Linux machine-name generation is unchanged. Hyper-V generation (Windows) - Request a Generation 2 Windows image from the RP, when the selected SKU supports it, via the UseWindowsGen2VM header. In Machine API mode the RP resolves the Windows image generation server-side and, for the Windows2022/Windows2019 OSSKUs, defaults to Generation 1 unless this header is set (it then rejects the create if the SKU does not support the requested generation). Karpenter selects the cheapest compatible SKU, which is frequently Gen2-only, so it sets the header exactly when the chosen SKU advertises Gen2 (karpenter.azure.com/sku-hyperv-generation): Gen2-only and dual-generation SKUs get Gen2 (preferred), Gen1-only SKUs fall back to the RP's Gen1 default. This mirrors the gen2-then-gen1 preference of the in-provider Windows image family and lets Windows NodePools provision on any SKU generation. The header is threaded through both the standard and the header-batch create paths (and participates in the batch key). Hybrid clusters - Always pin the Karpenter controller Deployment to Linux nodes (kubernetes.io/os=linux), merged with any user-provided nodeSelector and taking precedence over it. Now that Karpenter can provision Windows nodes, this guarantees the Linux-only controller is never scheduled onto a Windows node in a mixed-OS cluster. Tests and tooling - Unit tests for the OS/os-sku mapping, case-insensitive family matching, the Windows image family and GetImageFamily wiring, Windows node-image-version conversion, OSSKU/OSType selection in the Machine builder, the AKSWindows gallery filter, the pool-aware Windows machine-name generator, and the Gen2-image (UseWindowsGen2VM) decision and its batch-key/header plumbing. - New e2e suite (test/suites/windows) that provisions a Windows node via the Machine API and runs a Windows pod, asserting the node's os and os-sku labels. It runs in Machine API mode against either the reserved NAP-managed agent pool or a custom machines pool whose name is <=6 chars (the pool-aware Windows machine-name budget). The Windows NodePool's SKU is intentionally left unconstrained: because Karpenter requests a Gen2 image whenever the selected SKU supports it, Windows provisions on any Hyper-V generation, including Gen2-only sizes. Adds WindowsNodeClass/WindowsNodePool helpers, an az-mkaks-windows cluster target with a windowsProfile, an AKSWindows SIG reader role assignment, and makes the self-hosted machines-pool name configurable. Scope and notes - Windows is amd64-only by design. - Windows support targets the AKS Machine API path only; the in-provider bootstrapping paths (aksscriptless/bootstrappingclient) continue to reject Windows. - GPU-on-Windows and Windows-specific max-pods defaults are intentionally left for follow-up. CI - Wire the Windows e2e suite into the E2E matrix so it runs daily and on push. Windows is special-cased in workflows/e2e.yaml: it always runs in AKS Machine API mode (it is only provisionable that way and otherwise skips) on a dedicated cluster (new ci-mkcluster-all-windows target -> az-mkaks-windows, Azure CNI overlay + windowsProfile) because Windows does not support the Cilium dataplane used by the default CI cluster, with a machines pool name <= 6 chars (winmp) for the Windows machine-name limit. The ephemeral cluster's Windows admin password is generated and masked in the create-cluster action (no new repository secret). Non-Windows suites are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../actions/e2e/create-cluster/action.yaml | 21 ++- .../actions/e2e/install-karpenter/action.yaml | 5 + .github/workflows/README.md | 8 + .github/workflows/e2e-matrix.yaml | 1 + .github/workflows/e2e.yaml | 14 +- Makefile-az.mk | 32 ++++ README.md | 2 +- .../karpenter.azure.com_aksnodeclasses.yaml | 8 + charts/karpenter/templates/deployment.yaml | 9 +- charts/karpenter/values.yaml | 4 +- .../karpenter.azure.com_aksnodeclasses.yaml | 8 + pkg/apis/v1beta1/aksnodeclass.go | 4 +- pkg/apis/v1beta1/labels.go | 55 ++++++- pkg/apis/v1beta1/labels_test.go | 36 +++++ pkg/consts/consts.go | 7 + pkg/fake/nodeimageversionsapi_test.go | 14 +- .../aksmachinesheaderbatch/batchkey.go | 5 +- .../aksmachinesheaderbatch/batchkey_test.go | 17 ++ .../azclient/aksmachinesheaderbatch/client.go | 7 +- .../aksmachinesheaderbatch/executor.go | 17 +- .../aksmachinesheaderbatch/executor_test.go | 37 ++++- .../aksmachinesheaderbatch/fakeheader.go | 19 +++ pkg/providers/imagefamily/consts.go | 4 + .../imagefamily/nodeimageversionsclient.go | 4 +- pkg/providers/imagefamily/resolver.go | 3 + pkg/providers/imagefamily/windows.go | 151 ++++++++++++++++++ pkg/providers/imagefamily/windows_test.go | 96 +++++++++++ pkg/providers/instance/aksmachineinstance.go | 47 +++++- .../instance/aksmachineinstancehelpers.go | 124 +++++++++----- .../aksmachineinstancehelpers_test.go | 90 +++++++++++ .../instance/aksmachineinstanceutils.go | 54 ++++++- .../instance/aksmachineinstanceutils_test.go | 88 ++++++++-- pkg/providers/instancetype/instancetype.go | 2 +- pkg/utils/image.go | 20 ++- pkg/utils/image_test.go | 18 +++ pkg/utils/utils.go | 3 +- test/pkg/environment/azure/environment.go | 26 ++- test/suites/windows/suite_test.go | 128 +++++++++++++++ 38 files changed, 1096 insertions(+), 92 deletions(-) create mode 100644 pkg/providers/imagefamily/windows.go create mode 100644 pkg/providers/imagefamily/windows_test.go create mode 100644 test/suites/windows/suite_test.go diff --git a/.github/actions/e2e/create-cluster/action.yaml b/.github/actions/e2e/create-cluster/action.yaml index 53cee8f973..2048b22e55 100644 --- a/.github/actions/e2e/create-cluster/action.yaml +++ b/.github/actions/e2e/create-cluster/action.yaml @@ -33,6 +33,14 @@ inputs: description: "the azure vm size to use for the e2e test (set to empty string to allow AKS to default)" required: false default: "" + windows: + description: "When 'true', create a dedicated Windows-capable cluster (Azure CNI overlay + windowsProfile) instead of the default Cilium cluster" + required: false + default: "false" + aks_machines_pool_name: + description: "Name of the AKS 'machines' mode agent pool to create (machine API modes only). Windows requires a name <= 6 chars." + required: false + default: "testmpool" runs: using: "composite" steps: @@ -50,8 +58,19 @@ runs: AZURE_VM_SIZE: ${{ inputs.azure_vm_size }} K8S_VERSION: ${{ inputs.k8s_version }} PROVISION_MODE: ${{ inputs.provision_mode }} + AKS_MACHINES_POOL_NAME: ${{ inputs.aks_machines_pool_name }} run: | - if [ "${{ inputs.identity_type }}" = "UserAssigned" ]; then + if [ "${{ inputs.windows }}" = "true" ]; then + echo "Creating dedicated Windows-capable cluster (Azure CNI overlay + windowsProfile)" + # Generate a throwaway Windows admin password for this ephemeral cluster (meets Windows + # complexity: upper, lower, digit, special). Mask it so it never appears in logs. The RP + # sources Windows node admin credentials from the cluster's windowsProfile; the tests + # never use it and the cluster is deleted at the end of the run. + WINDOWS_ADMIN_PASSWORD="Aks$(openssl rand -base64 24 | tr -dc 'A-Za-z0-9' | head -c 24)9!" + echo "::add-mask::$WINDOWS_ADMIN_PASSWORD" + export WINDOWS_ADMIN_PASSWORD + make ci-mkcluster-all-windows + elif [ "${{ inputs.identity_type }}" = "UserAssigned" ]; then echo "Creating cluster with user-assigned managed identity" make ci-mkcluster-all-userassigned else diff --git a/.github/actions/e2e/install-karpenter/action.yaml b/.github/actions/e2e/install-karpenter/action.yaml index 52c578c362..db55ec132d 100644 --- a/.github/actions/e2e/install-karpenter/action.yaml +++ b/.github/actions/e2e/install-karpenter/action.yaml @@ -22,6 +22,10 @@ inputs: provision_mode: description: "the Karpenter provisioning mode to run the e2e test in" default: "aksscriptless" + aks_machines_pool_name: + description: "Name of the AKS 'machines' mode agent pool (machine API modes only). Must match the pool created at cluster-create time." + required: false + default: "testmpool" runs: using: "composite" steps: @@ -35,6 +39,7 @@ runs: AZURE_CLUSTER_NAME: ${{ inputs.cluster_name }} AZURE_LOCATION: ${{ inputs.location }} PROVISION_MODE: ${{ inputs.provision_mode }} + AKS_MACHINES_POOL_NAME: ${{ inputs.aks_machines_pool_name }} AZURE_ACR_NAME: ${{ inputs.acr_name }} # Redirect Go temp/cache to /mnt/ (Azure temporary disk) which has more disk # space than the OS disk. 1ES pool VMs can run out of disk on / during large builds. diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 851756e28d..fec253af90 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -7,6 +7,14 @@ 1. Create your new E2E testing suite `` within the `test/suites/` package. See: `test/README.md` 2. Update the `workflows/e2e-matrix.yaml` workflow to include your E2E test case: `suite: [Utilization, GPU, ...]` - add in the name of your folder within the `test/suites/` package to the comma separated list. Casing does not matter. +> **Note — suites that need a non-default cluster:** most suites run on the shared CI cluster +> (`ci-mkcluster-all`, Azure CNI overlay + Cilium, in the matrix's `provision_mode`). The `Windows` +> suite is special-cased in `workflows/e2e.yaml`: it always runs in `aksmachineapi` mode (Windows is +> only provisionable via the AKS Machine API) on a dedicated cluster (`ci-mkcluster-all-windows`, +> `az-mkaks-windows`) because Windows does not support the Cilium dataplane, and it uses a machines +> pool name `<= 6` chars (`winmp`) to satisfy the Windows machine-name limit. Follow that pattern if a +> new suite needs its own cluster shape or provisioning mode. + ### Running the test case (temporary workflow until we re-enable automation) diff --git a/.github/workflows/e2e-matrix.yaml b/.github/workflows/e2e-matrix.yaml index 91ee183182..d1613e4b87 100644 --- a/.github/workflows/e2e-matrix.yaml +++ b/.github/workflows/e2e-matrix.yaml @@ -51,6 +51,7 @@ jobs: - Storage - Subnet - Utilization + - Windows permissions: contents: read statuses: write diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml index 86feaa594a..15289e5d4c 100644 --- a/.github/workflows/e2e.yaml +++ b/.github/workflows/e2e.yaml @@ -51,6 +51,7 @@ on: - Storage - Subnet - Utilization + - Windows location: type: choice description: "Azure location to run the e2e test in" @@ -216,9 +217,14 @@ jobs: git_ref: ${{ inputs.git_ref }} location: ${{ inputs.location }} k8s_version: ${{ inputs.k8s_version }} - provision_mode: ${{ inputs.provision_mode }} + # Windows is only provisionable via the AKS Machine API, so force that mode for the Windows + # suite regardless of the matrix's provision_mode (it would otherwise skip under aksscriptless). + provision_mode: ${{ inputs.suite == 'Windows' && 'aksmachineapi' || inputs.provision_mode }} identity_type: ${{ inputs.suite == 'Machine' && 'UserAssigned' || 'SystemAssigned' }} azure_vm_size: ${{ inputs.azure_vm_size }} + # Windows needs a dedicated (non-Cilium) cluster and a machines pool name <= 6 chars. + windows: ${{ inputs.suite == 'Windows' }} + aks_machines_pool_name: ${{ inputs.suite == 'Windows' && 'winmp' || 'testmpool' }} - name: install karpenter uses: ./.github/actions/e2e/install-karpenter with: @@ -228,7 +234,8 @@ jobs: acr_name: ${{ env.ACR_NAME }} git_ref: ${{ inputs.git_ref }} location: ${{ inputs.location }} - provision_mode: ${{ inputs.provision_mode }} + provision_mode: ${{ inputs.suite == 'Windows' && 'aksmachineapi' || inputs.provision_mode }} + aks_machines_pool_name: ${{ inputs.suite == 'Windows' && 'winmp' || 'testmpool' }} - name: run the ${{ inputs.suite }} test suite env: AZURE_CLUSTER_NAME: ${{ env.CLUSTER_NAME }} @@ -239,7 +246,8 @@ jobs: AZURE_CLIENT_ID: ${{ secrets.E2E_CLIENT_ID_TEST }} TEST_SUITE: ${{ inputs.suite }} GIT_REF: ${{ github.sha }} - PROVISION_MODE: ${{ inputs.provision_mode }} + PROVISION_MODE: ${{ inputs.suite == 'Windows' && 'aksmachineapi' || inputs.provision_mode }} + AKS_MACHINES_POOL_NAME: ${{ inputs.suite == 'Windows' && 'winmp' || 'testmpool' }} run: | make az-creds make e2etests diff --git a/Makefile-az.mk b/Makefile-az.mk index dbfd0e7fd9..7dc60f388d 100755 --- a/Makefile-az.mk +++ b/Makefile-az.mk @@ -37,6 +37,12 @@ KO_BASE_IMAGE_AMD64 ?= mcr.microsoft.com/azurelinux/distroless/base@sha256:301f0 KO_BASE_IMAGE_ARM64 ?= mcr.microsoft.com/azurelinux/distroless/base@sha256:ef54cbe5a632f71090688f45901d073f19f414eb38516a60891ce3dff33c2029 export KOCACHE ?= $(or $(RUNNER_TEMP),/tmp)/ko-cache +# Windows admin credentials, used when creating a Windows-capable test cluster +# (az-mkaks-windows). The AKS RP sources Windows node admin credentials from the cluster's +# windowsProfile, so the cluster must be created with these. Override for non-throwaway use. +WINDOWS_ADMIN_USERNAME ?= azureuser +WINDOWS_ADMIN_PASSWORD ?= Repl@ceMe-W1ndows-E2E! + .DEFAULT_GOAL := help # make without arguments will show help export KO_GO_PATH ?= hack/go-crossbuild.sh @@ -120,6 +126,13 @@ ci-mkcluster-all: az-create-workload-msi az-mkaks-cilium ci-mkcluster-all-userassigned: az-create-workload-msi az-mkaks-cilium-userassigned az-create-federated-cred $(AZ_ALL_PERMS) +# Windows e2e needs a dedicated cluster: Windows does not support the Cilium dataplane used by the +# default CI cluster, so this uses az-mkaks-windows (Azure CNI overlay + windowsProfile). The Windows +# suite only provisions via the AKS Machine API, so invoke with PROVISION_MODE=aksmachineapi (which +# pulls the machine perms + machines pool into AZ_ALL_PERMS) and AKS_MACHINES_POOL_NAME set to a name +# <= 6 chars (Windows machine-name budget for a custom pool). +ci-mkcluster-all-windows: az-create-workload-msi az-mkaks-windows az-create-federated-cred $(AZ_ALL_PERMS) + ci-install: az-configure-values az-build az-run # --------------------------------------------- @@ -233,6 +246,24 @@ az-mkaks-overlay: az-mkacr ## Create test AKS cluster (with --network-plugin-mod $(MAKE) az-creds skaffold config set default-repo $(AZURE_ACR_NAME).$(AZURE_ACR_SUFFIX)/karpenter +az-mkaks-windows: az-mkacr ## Create a Windows-capable test AKS cluster (Azure CNI overlay + windowsProfile) for the Windows e2e suite + @hack/deploy/check-cluster-exists.sh $(AZURE_CLUSTER_NAME) $(AZURE_RESOURCE_GROUP) az-mkaks-windows; \ + EXIT_CODE=$$?; \ + if [ $$EXIT_CODE -eq 1 ]; then \ + az aks create --name $(AZURE_CLUSTER_NAME) --resource-group $(AZURE_RESOURCE_GROUP) --attach-acr $(AZURE_ACR_NAME) \ + --enable-managed-identity --node-count 3 --generate-ssh-keys \ + --network-plugin azure --network-plugin-mode overlay \ + --windows-admin-username $(WINDOWS_ADMIN_USERNAME) --windows-admin-password '$(WINDOWS_ADMIN_PASSWORD)' \ + --enable-oidc-issuer --enable-workload-identity --nodepool-taints "CriticalAddonsOnly=true:NoSchedule" \ + $(if $(AZURE_VM_SIZE),--node-vm-size $(AZURE_VM_SIZE)) \ + $(if $(K8S_VERSION),--kubernetes-version $(K8S_VERSION)) \ + --tags "make-command=az-mkaks-windows"; \ + elif [ $$EXIT_CODE -eq 2 ]; then \ + exit 1; \ + fi + $(MAKE) az-creds + skaffold config set default-repo $(AZURE_ACR_NAME).$(AZURE_ACR_SUFFIX)/karpenter + az-mkaks-perftest: az-mkacr ## Create test AKS cluster (with Azure Overlay, larger system pool VMs and larger pod-cidr) @hack/deploy/check-cluster-exists.sh $(AZURE_CLUSTER_NAME) $(AZURE_RESOURCE_GROUP) az-mkaks-perftest; \ EXIT_CODE=$$?; \ @@ -318,6 +349,7 @@ az-perm-sig: ## Create role assignments when testing with SIG images $(eval KARPENTER_USER_ASSIGNED_CLIENT_ID=$(shell az identity show --resource-group "${AZURE_RESOURCE_GROUP}" --name "${AZURE_KARPENTER_USER_ASSIGNED_IDENTITY_NAME}" --query 'principalId' --output tsv)) az role assignment create --assignee-object-id $(KARPENTER_USER_ASSIGNED_CLIENT_ID) --assignee-principal-type "ServicePrincipal" --role "Reader" --scope /subscriptions/$(AZURE_SIG_SUBSCRIPTION_ID)/resourceGroups/AKS-Ubuntu/providers/Microsoft.Compute/galleries/AKSUbuntu az role assignment create --assignee-object-id $(KARPENTER_USER_ASSIGNED_CLIENT_ID) --assignee-principal-type "ServicePrincipal" --role "Reader" --scope /subscriptions/$(AZURE_SIG_SUBSCRIPTION_ID)/resourceGroups/AKS-AzureLinux/providers/Microsoft.Compute/galleries/AKSAzureLinux + az role assignment create --assignee-object-id $(KARPENTER_USER_ASSIGNED_CLIENT_ID) --assignee-principal-type "ServicePrincipal" --role "Reader" --scope /subscriptions/$(AZURE_SIG_SUBSCRIPTION_ID)/resourceGroups/AKS-Windows/providers/Microsoft.Compute/galleries/AKSWindows az-perm-subnet-custom: az-perm ## Create role assignments to let Karpenter manage VMs and Network (custom VNet) $(eval VNET_SUBNET_ID=$(shell az aks show --name $(AZURE_CLUSTER_NAME) --resource-group $(AZURE_RESOURCE_GROUP) --query "agentPoolProfiles[0].vnetSubnetId" --output tsv)) diff --git a/README.md b/README.md index 9184260e65..08cf7bdd31 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Karpenter provider for AKS can be used in two modes: ## Known limitations The following AKS features are not supported: -* Windows nodes. +* Windows nodes in self-hosted mode. Windows node provisioning is supported only with the AKS Machine API provision mode (used by Node Auto Provisioning); the self-hosted (scriptless) provision mode does not support Windows. * Kubenet and Calico. * IPv6 clusters. * [Service Principal](https://learn.microsoft.com/azure/aks/kubernetes-service-principal) based clusters. A system-assigned or user-assigned managed identity must be used. diff --git a/charts/karpenter-crd/templates/karpenter.azure.com_aksnodeclasses.yaml b/charts/karpenter-crd/templates/karpenter.azure.com_aksnodeclasses.yaml index fe560ce078..d7d6d554f8 100644 --- a/charts/karpenter-crd/templates/karpenter.azure.com_aksnodeclasses.yaml +++ b/charts/karpenter-crd/templates/karpenter.azure.com_aksnodeclasses.yaml @@ -1005,6 +1005,8 @@ spec: - Ubuntu2204 - Ubuntu2404 - AzureLinux + - Windows2022 + - Windows2025 type: string kubelet: description: |- @@ -1680,6 +1682,12 @@ spec: rule: 'has(self.fipsMode) && self.fipsMode == ''FIPS'' ? (has(self.imageFamily) && self.imageFamily != ''Ubuntu2204'' && self.imageFamily != ''Ubuntu2404'') : true' + - message: FIPS is not supported for Windows image families + rule: '!has(self.fipsMode) || self.fipsMode != ''FIPS'' || !has(self.imageFamily) + || !(self.imageFamily in [''Windows2022'',''Windows2025''])' + - message: linuxOSConfig is not supported for Windows image families + rule: '!has(self.linuxOSConfig) || !has(self.imageFamily) || !(self.imageFamily + in [''Windows2022'',''Windows2025''])' - message: kubelet.failSwapOn must be set to false when linuxOSConfig.swapFileSize is specified rule: '!has(self.linuxOSConfig) || !has(self.linuxOSConfig.swapFileSize) diff --git a/charts/karpenter/templates/deployment.yaml b/charts/karpenter/templates/deployment.yaml index 1d808fc8c6..9d63d4901a 100644 --- a/charts/karpenter/templates/deployment.yaml +++ b/charts/karpenter/templates/deployment.yaml @@ -239,10 +239,13 @@ spec: initContainers: {{- toYaml . | nindent 8 }} {{- end }} - {{- with .Values.nodeSelector }} + {{- /* The Karpenter controller is a Linux-only component. Always pin it to Linux + nodes so it is never scheduled onto Windows nodes in hybrid clusters, while + still honoring any additional user-provided nodeSelector entries. */}} nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} + {{- $nodeSelector := merge (dict) (.Values.nodeSelector | default dict) }} + {{- $_ := set $nodeSelector "kubernetes.io/os" "linux" }} + {{- toYaml $nodeSelector | nindent 8 }} {{- with .Values.affinity }} # The template below patches the .Values.affinity to add a default label selector where not specified {{- $_ := include "karpenter.patchAffinity" $ }} diff --git a/charts/karpenter/values.yaml b/charts/karpenter/values.yaml index 921fbf636c..6c19d6b611 100644 --- a/charts/karpenter/values.yaml +++ b/charts/karpenter/values.yaml @@ -81,7 +81,9 @@ dnsConfig: {} # options: # - name: ndots # value: "1" -# -- Node selectors to schedule the pod to nodes with labels. +# -- Node selectors to schedule the pod to nodes with labels. Note: the controller is a +# Linux-only component, so `kubernetes.io/os: linux` is always enforced by the chart (even +# if overridden here) to keep it off Windows nodes in hybrid clusters. nodeSelector: kubernetes.io/os: linux # -- Affinity rules for scheduling the pod. If an explicit label selector is not provided for pod affinity or pod anti-affinity one will be created from the pod selector labels. diff --git a/pkg/apis/crds/karpenter.azure.com_aksnodeclasses.yaml b/pkg/apis/crds/karpenter.azure.com_aksnodeclasses.yaml index fe560ce078..d7d6d554f8 100644 --- a/pkg/apis/crds/karpenter.azure.com_aksnodeclasses.yaml +++ b/pkg/apis/crds/karpenter.azure.com_aksnodeclasses.yaml @@ -1005,6 +1005,8 @@ spec: - Ubuntu2204 - Ubuntu2404 - AzureLinux + - Windows2022 + - Windows2025 type: string kubelet: description: |- @@ -1680,6 +1682,12 @@ spec: rule: 'has(self.fipsMode) && self.fipsMode == ''FIPS'' ? (has(self.imageFamily) && self.imageFamily != ''Ubuntu2204'' && self.imageFamily != ''Ubuntu2404'') : true' + - message: FIPS is not supported for Windows image families + rule: '!has(self.fipsMode) || self.fipsMode != ''FIPS'' || !has(self.imageFamily) + || !(self.imageFamily in [''Windows2022'',''Windows2025''])' + - message: linuxOSConfig is not supported for Windows image families + rule: '!has(self.linuxOSConfig) || !has(self.imageFamily) || !(self.imageFamily + in [''Windows2022'',''Windows2025''])' - message: kubelet.failSwapOn must be set to false when linuxOSConfig.swapFileSize is specified rule: '!has(self.linuxOSConfig) || !has(self.linuxOSConfig.swapFileSize) diff --git a/pkg/apis/v1beta1/aksnodeclass.go b/pkg/apis/v1beta1/aksnodeclass.go index ce299caf16..c25d9d998b 100644 --- a/pkg/apis/v1beta1/aksnodeclass.go +++ b/pkg/apis/v1beta1/aksnodeclass.go @@ -61,6 +61,8 @@ func (a *ArtifactStreaming) IsEnabled(arch string) bool { // AKSNodeClassSpec is the top level specification for the AKS Karpenter Provider. // This will contain configuration necessary to launch instances in AKS. // +kubebuilder:validation:XValidation:message="FIPS is not yet supported for Ubuntu2204 or Ubuntu2404",rule="has(self.fipsMode) && self.fipsMode == 'FIPS' ? (has(self.imageFamily) && self.imageFamily != 'Ubuntu2204' && self.imageFamily != 'Ubuntu2404') : true" +// +kubebuilder:validation:XValidation:message="FIPS is not supported for Windows image families",rule="!has(self.fipsMode) || self.fipsMode != 'FIPS' || !has(self.imageFamily) || !(self.imageFamily in ['Windows2022','Windows2025'])" +// +kubebuilder:validation:XValidation:message="linuxOSConfig is not supported for Windows image families",rule="!has(self.linuxOSConfig) || !has(self.imageFamily) || !(self.imageFamily in ['Windows2022','Windows2025'])" // +kubebuilder:validation:XValidation:message="kubelet.failSwapOn must be set to false when linuxOSConfig.swapFileSize is specified",rule="!has(self.linuxOSConfig) || !has(self.linuxOSConfig.swapFileSize) || (has(self.kubelet) && has(self.kubelet.failSwapOn) && self.kubelet.failSwapOn == false)" type AKSNodeClassSpec struct { // vnetSubnetID is the subnet used by nics provisioned with this nodeclass. @@ -79,7 +81,7 @@ type AKSNodeClassSpec struct { ImageID *string `json:"-"` // imageFamily is the image family that instances use. // +default="Ubuntu" - // +kubebuilder:validation:Enum:={Ubuntu,Ubuntu2204,Ubuntu2404,AzureLinux} + // +kubebuilder:validation:Enum:={Ubuntu,Ubuntu2204,Ubuntu2404,AzureLinux,Windows2022,Windows2025} // +optional ImageFamily *string `json:"imageFamily,omitempty"` // fipsMode controls FIPS compliance for the provisioned nodes diff --git a/pkg/apis/v1beta1/labels.go b/pkg/apis/v1beta1/labels.go index dbb5e5d532..0b6303033f 100644 --- a/pkg/apis/v1beta1/labels.go +++ b/pkg/apis/v1beta1/labels.go @@ -42,7 +42,7 @@ func init() { karpv1.WellKnownValuesForRequirements[AKSLabelMode] = sets.New(ModeSystem, ModeUser) karpv1.WellKnownValuesForRequirements[AKSLabelScaleSetPriority] = sets.New(ScaleSetPriorityRegular, ScaleSetPrioritySpot) karpv1.WellKnownValuesForRequirements[AKSLabelPriority] = sets.New(PriorityRegular, PrioritySpot) - karpv1.WellKnownValuesForRequirements[AKSLabelOSSKU] = sets.New(OSSKUUbuntu, OSSKUAzureLinux) + karpv1.WellKnownValuesForRequirements[AKSLabelOSSKU] = sets.New(OSSKUUbuntu, OSSKUAzureLinux, OSSKUWindows2022, OSSKUWindows2025) karpv1.WellKnownValuesForRequirements[AKSLabelFIPSEnabled] = sets.New("true") } @@ -190,11 +190,26 @@ const ( Ubuntu2204ImageFamily = "Ubuntu2204" Ubuntu2404ImageFamily = "Ubuntu2404" AzureLinuxImageFamily = "AzureLinux" + + // Windows image families. Each maps 1:1 to an AKS Windows OSSKU. + Windows2022ImageFamily = "Windows2022" + Windows2025ImageFamily = "Windows2025" ) const ( OSSKUUbuntu = "Ubuntu" OSSKUAzureLinux = "AzureLinux" + + // Windows os-sku label values. These match the kubernetes.azure.com/os-sku + // values AKS applies to Windows nodes (version-specific, unlike Ubuntu which collapses). + OSSKUWindows2022 = "Windows2022" + OSSKUWindows2025 = "Windows2025" +) + +// OS label values for kubernetes.io/os (and the AKS Machine OSType mapping). +const ( + OSLinux = "linux" + OSWindows = "windows" ) const ( @@ -213,13 +228,21 @@ var UbuntuFamilies = sets.New( Ubuntu2404ImageFamily, ) +// WindowsFamilies is the set of imageFamily spec values that provision Windows nodes. +var WindowsFamilies = sets.New( + Windows2022ImageFamily, + Windows2025ImageFamily, +) + // imageFamilyToOSSKU maps imageFamily spec values to os-sku label values. // These values match what AKS writes for kubernetes.azure.com/os-sku. var imageFamilyToOSSKU = map[string]string{ - UbuntuImageFamily: OSSKUUbuntu, - Ubuntu2204ImageFamily: OSSKUUbuntu, - Ubuntu2404ImageFamily: OSSKUUbuntu, - AzureLinuxImageFamily: OSSKUAzureLinux, + UbuntuImageFamily: OSSKUUbuntu, + Ubuntu2204ImageFamily: OSSKUUbuntu, + Ubuntu2404ImageFamily: OSSKUUbuntu, + AzureLinuxImageFamily: OSSKUAzureLinux, + Windows2022ImageFamily: OSSKUWindows2022, + Windows2025ImageFamily: OSSKUWindows2025, } // GetOSSKUFromImageFamily returns the kuberentes.azure.com/os-sku label value for the given imageFamily. @@ -238,3 +261,25 @@ func GetOSSKUFromImageFamily(imageFamily string) string { func IsAKSLabel(label string) bool { return strings.HasPrefix(label, AKSLabelDomain+"/") || aksLegacyLabels.Has(label) } + +// GetOSForImageFamily returns the kubernetes.io/os value ("linux" or "windows") for the +// given imageFamily. Windows families return "windows"; everything else (including empty, +// which defaults to Ubuntu) returns "linux". +// IsWindowsImageFamily reports whether the given imageFamily provisions Windows nodes. +// The comparison is case-insensitive for robustness; canonical values (enforced by the +// AKSNodeClass imageFamily CEL enum) are the Windows*ImageFamily constants. +func IsWindowsImageFamily(imageFamily string) bool { + for family := range WindowsFamilies { + if strings.EqualFold(imageFamily, family) { + return true + } + } + return false +} + +func GetOSForImageFamily(imageFamily string) string { + if IsWindowsImageFamily(imageFamily) { + return OSWindows + } + return OSLinux +} diff --git a/pkg/apis/v1beta1/labels_test.go b/pkg/apis/v1beta1/labels_test.go index b1673063dd..d2a96579f7 100644 --- a/pkg/apis/v1beta1/labels_test.go +++ b/pkg/apis/v1beta1/labels_test.go @@ -49,6 +49,16 @@ func TestGetOSSKUFromImageFamily(t *testing.T) { imageFamily: v1beta1.AzureLinuxImageFamily, expected: "AzureLinux", }, + { + name: "Windows2022", + imageFamily: v1beta1.Windows2022ImageFamily, + expected: "Windows2022", + }, + { + name: "Windows2025", + imageFamily: v1beta1.Windows2025ImageFamily, + expected: "Windows2025", + }, { name: "empty string defaults to Ubuntu", imageFamily: "", @@ -69,3 +79,29 @@ func TestGetOSSKUFromImageFamily(t *testing.T) { }) } } + +func TestGetOSForImageFamily(t *testing.T) { + cases := []struct { + name string + imageFamily string + expected string + }{ + {name: "Ubuntu default", imageFamily: v1beta1.UbuntuImageFamily, expected: "linux"}, + {name: "Ubuntu2204", imageFamily: v1beta1.Ubuntu2204ImageFamily, expected: "linux"}, + {name: "Ubuntu2404", imageFamily: v1beta1.Ubuntu2404ImageFamily, expected: "linux"}, + {name: "AzureLinux", imageFamily: v1beta1.AzureLinuxImageFamily, expected: "linux"}, + {name: "empty string defaults to linux", imageFamily: "", expected: "linux"}, + {name: "unknown family defaults to linux", imageFamily: "CustomOS", expected: "linux"}, + {name: "Windows2022", imageFamily: v1beta1.Windows2022ImageFamily, expected: "windows"}, + {name: "Windows2025", imageFamily: v1beta1.Windows2025ImageFamily, expected: "windows"}, + {name: "Windows is case-insensitive", imageFamily: "windows2022", expected: "windows"}, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + g := NewWithT(t) + result := v1beta1.GetOSForImageFamily(c.imageFamily) + g.Expect(result).To(Equal(c.expected)) + }) + } +} diff --git a/pkg/consts/consts.go b/pkg/consts/consts.go index a3670552fc..a782a64b84 100644 --- a/pkg/consts/consts.go +++ b/pkg/consts/consts.go @@ -48,6 +48,13 @@ const ( AKSMachineAPIHeaderBatchMaxSize = 50 + // HeaderUseWindowsGen2VM is the HTTP request header the AKS RP reads to decide whether a Windows + // node should be provisioned from a Generation 2 image. For the Windows2022/Windows2019 OSSKUs the + // RP defaults to a Generation 1 image and only selects Gen2 when this header is "true"; Windows2025 + // and the Annual channel pick the generation from the VM size automatically. See the AKS RP + // getDistroForWindows/validateVMSizeGen2Support logic. + HeaderUseWindowsGen2VM = "UseWindowsGen2VM" + // Provisioning states for AKS Machine objects. // The SDK's Machine.Properties.ProvisioningState is typed as *string (no typed constants). // Suggestion: find a constant from azure-sdk-for-go if one becomes available. diff --git a/pkg/fake/nodeimageversionsapi_test.go b/pkg/fake/nodeimageversionsapi_test.go index c7822bb5d2..4cba76c8f1 100644 --- a/pkg/fake/nodeimageversionsapi_test.go +++ b/pkg/fake/nodeimageversionsapi_test.go @@ -23,6 +23,7 @@ import ( "github.com/Azure/karpenter-provider-azure/pkg/providers/imagefamily" . "github.com/onsi/gomega" "github.com/samber/lo" + "k8s.io/apimachinery/pkg/util/sets" ) func TestFilteredNodeImagesGalleryFilter(t *testing.T) { @@ -30,10 +31,19 @@ func TestFilteredNodeImagesGalleryFilter(t *testing.T) { nodeImageVersionAPI := NodeImageVersionsAPI{} nodeImageVersions, _ := nodeImageVersionAPI.List(context.TODO(), "") filteredNodeImages := imagefamily.FilteredNodeImages(nodeImageVersions) + supportedGalleries := sets.New("AKSUbuntu", "AKSAzureLinux", "AKSWindows") + sawWindows := false for _, val := range filteredNodeImages { - g.Expect(lo.FromPtr(val.OS)).ToNot(Equal("AKSWindows")) - g.Expect(lo.FromPtr(val.OS)).ToNot(Equal("AKSUbuntuEdgeZone")) + os := lo.FromPtr(val.OS) + // Only supported galleries are returned; unsupported ones (e.g. EdgeZone) are filtered out. + g.Expect(supportedGalleries.Has(os)).To(BeTrue(), "unexpected gallery %q in filtered images", os) + g.Expect(os).ToNot(Equal("AKSUbuntuEdgeZone")) + if os == "AKSWindows" { + sawWindows = true + } } + // Windows is now a supported gallery and must survive filtering. + g.Expect(sawWindows).To(BeTrue(), "expected at least one AKSWindows image to be included") } // The reasoning behind the test is the following set of output diff --git a/pkg/providers/azclient/aksmachinesheaderbatch/batchkey.go b/pkg/providers/azclient/aksmachinesheaderbatch/batchkey.go index e60ac1edb9..2686b8a930 100644 --- a/pkg/providers/azclient/aksmachinesheaderbatch/batchkey.go +++ b/pkg/providers/azclient/aksmachinesheaderbatch/batchkey.go @@ -38,7 +38,10 @@ func determineBatchKey(item *aksMachineCreatePayload) (string, error) { } hash := sha256.Sum256(jsonBytes) - prefix := item.resourceGroupName + "/" + item.resourceName + "/" + item.agentPoolName + "/" + // useWindowsGen2VM is part of the key so machines that disagree on the requested Windows image + // generation never coalesce into one batch (they also differ in vmSize/OSSKU, so this is + // defensive). The executor reads the value from the first request when composing batch headers. + prefix := fmt.Sprintf("%s/%s/%s/gen2=%t/", item.resourceGroupName, item.resourceName, item.agentPoolName, item.useWindowsGen2VM) return prefix + fmt.Sprintf("%x", hash[:8]), nil } diff --git a/pkg/providers/azclient/aksmachinesheaderbatch/batchkey_test.go b/pkg/providers/azclient/aksmachinesheaderbatch/batchkey_test.go index 1d72744f56..ed9445e0b2 100644 --- a/pkg/providers/azclient/aksmachinesheaderbatch/batchkey_test.go +++ b/pkg/providers/azclient/aksmachinesheaderbatch/batchkey_test.go @@ -100,6 +100,23 @@ func TestMachineKeyFunc_ReadOnlyFieldsExcluded(t *testing.T) { g.Expect(mustDetermineBatchKey(t, &item2)).To(gomega.Equal(mustDetermineBatchKey(t, &item1)), "read-only fields should not affect hash") } +func TestMachineKeyFunc_UseWindowsGen2VMSeparatesBatches(t *testing.T) { + t.Parallel() + g := gomega.NewWithT(t) + + vmSize := "Standard_D2s_v3" + body := func() *armcontainerservice.Machine { + return &armcontainerservice.Machine{Properties: &armcontainerservice.MachineProperties{ + Hardware: &armcontainerservice.MachineHardwareProfile{VMSize: &vmSize}, + }} + } + gen1 := aksMachineCreatePayload{machineBody: body(), useWindowsGen2VM: false} + gen2 := aksMachineCreatePayload{machineBody: body(), useWindowsGen2VM: true} + + g.Expect(mustDetermineBatchKey(t, &gen2)).ToNot(gomega.Equal(mustDetermineBatchKey(t, &gen1)), + "machines requesting different Windows image generations must not share a batch") +} + // realisticMachineProps returns a fully-populated MachineProperties matching // production templates built by buildAKSMachineTemplate. func realisticMachineProps(vmSize, nodeClaimName string) *armcontainerservice.MachineProperties { diff --git a/pkg/providers/azclient/aksmachinesheaderbatch/client.go b/pkg/providers/azclient/aksmachinesheaderbatch/client.go index d896ec6c31..eb33e268a6 100644 --- a/pkg/providers/azclient/aksmachinesheaderbatch/client.go +++ b/pkg/providers/azclient/aksmachinesheaderbatch/client.go @@ -37,7 +37,10 @@ type AKSMachinesHeaderBatchAPI interface { // API errors, but nothing else). // - The fact that HandlableError is considered one of the "expected" states in this // context, just not ideal. Operational error, on the other hand, is more of a bug. - BeginCreateWithBatch(ctx context.Context, resourceGroupName string, resourceName string, agentPoolName string, aksMachineName string, machine *armcontainerservice.Machine) (*offerings.HandlableError, error) + // + // useWindowsGen2VM requests a Generation 2 Windows image from the RP (UseWindowsGen2VM header). + // It participates in the batch key, so only machines that agree on it coalesce into one request. + BeginCreateWithBatch(ctx context.Context, resourceGroupName string, resourceName string, agentPoolName string, aksMachineName string, machine *armcontainerservice.Machine, useWindowsGen2VM bool) (*offerings.HandlableError, error) } // We don't need the rest of machine API interface. Just create. @@ -76,6 +79,7 @@ func (c *Client) BeginCreateWithBatch( agentPoolName string, machineName string, machine *armcontainerservice.Machine, + useWindowsGen2VM bool, ) (*offerings.HandlableError, error) { responseChan, err := c.b.Enqueue(aksMachineCreatePayload{ resourceGroupName: resourceGroupName, @@ -83,6 +87,7 @@ func (c *Client) BeginCreateWithBatch( agentPoolName: agentPoolName, machineName: machineName, machineBody: machine, + useWindowsGen2VM: useWindowsGen2VM, }) if err != nil { return nil, err diff --git a/pkg/providers/azclient/aksmachinesheaderbatch/executor.go b/pkg/providers/azclient/aksmachinesheaderbatch/executor.go index e6ebf55fa6..bf85e90e2b 100644 --- a/pkg/providers/azclient/aksmachinesheaderbatch/executor.go +++ b/pkg/providers/azclient/aksmachinesheaderbatch/executor.go @@ -23,6 +23,7 @@ import ( "github.com/Azure/azure-sdk-for-go/sdk/azcore/policy" "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9" + "github.com/Azure/karpenter-provider-azure/pkg/consts" "github.com/Azure/karpenter-provider-azure/pkg/providers/instance/offerings" "github.com/Azure/karpenter-provider-azure/pkg/utils/batcher" "sigs.k8s.io/controller-runtime/pkg/log" @@ -35,6 +36,10 @@ type aksMachineCreatePayload struct { agentPoolName string machineName string machineBody *armcontainerservice.Machine + // useWindowsGen2VM requests a Generation 2 Windows image from the RP for this machine. All + // requests in a batch share the same value (it is part of the batch key), so it is read from + // the first request when composing the batch's HTTP headers. + useWindowsGen2VM bool } // executor sends batches to Azure using the BatchPutMachine HTTP header. @@ -62,12 +67,20 @@ func (e *executor) executeBatch(ctx context.Context, batch *batcher.Batch[aksMac distributeOperationalError(batch, fmt.Errorf("failed to build header for batch API: %w", err)) return } - ctxWithHeader := policy.WithHTTPHeader(ctx, http.Header{ + batchHeaders := http.Header{ "BatchPutMachine": []string{header}, - }) + } + // All requests in a batch share the same useWindowsGen2VM value (it is part of the batch key), + // so request a Gen2 Windows image for the whole batch when the first request asks for it. + useWindowsGen2VM := batch.Requests[0].Payload.useWindowsGen2VM + if useWindowsGen2VM { + batchHeaders.Set(consts.HeaderUseWindowsGen2VM, "true") + } + ctxWithHeader := policy.WithHTTPHeader(ctx, batchHeaders) // Also mirror entries into context for fakes/testing. // See WithFakeBatchEntries for why this duplication is necessary. ctxWithHeader = WithFakeBatchEntries(ctxWithHeader, entries) + ctxWithHeader = WithFakeUseWindowsGen2VM(ctxWithHeader, useWindowsGen2VM) // Use resource params from the first request (all requests in a batch // share the same resource path due to the key function). diff --git a/pkg/providers/azclient/aksmachinesheaderbatch/executor_test.go b/pkg/providers/azclient/aksmachinesheaderbatch/executor_test.go index 8ff6508270..f63f92a618 100644 --- a/pkg/providers/azclient/aksmachinesheaderbatch/executor_test.go +++ b/pkg/providers/azclient/aksmachinesheaderbatch/executor_test.go @@ -82,7 +82,6 @@ func (r *recordingClient) snapshot() []recordedCall { // test helpers // --------------------------------------------------------------------------- -//nolint:unparam // vmSize is always the same today but kept as param for future test flexibility func tpl(vmSize string, zones []string, tags map[string]string) *armcontainerservice.Machine { m := &armcontainerservice.Machine{ Properties: &armcontainerservice.MachineProperties{ @@ -114,6 +113,12 @@ func makeReq(name string, template *armcontainerservice.Machine) *batcher.Batche } } +func makeReqGen2(name string, template *armcontainerservice.Machine, useWindowsGen2VM bool) *batcher.BatchedRequest[aksMachineCreatePayload, *offerings.HandlableError] { + req := makeReq(name, template) + req.Payload.useWindowsGen2VM = useWindowsGen2VM + return req +} + func makeBatch(requests ...*batcher.BatchedRequest[aksMachineCreatePayload, *offerings.HandlableError]) *batcher.Batch[aksMachineCreatePayload, *offerings.HandlableError] { if len(requests) == 0 { return &batcher.Batch[aksMachineCreatePayload, *offerings.HandlableError]{} @@ -202,6 +207,34 @@ func TestExecutorAttachesPerMachineEntriesToContext(t *testing.T) { g.Expect(entries[1].Tags).To(gomega.Equal(map[string]string{"b": "2"})) } +// TestExecutorAttachesUseWindowsGen2VMToContext verifies the executor requests a Gen2 Windows image +// for the batch exactly when the (first) request asks for it. The real UseWindowsGen2VM HTTP header +// is set via policy.WithHTTPHeader (unreadable in-process), so we assert the mirrored context value. +func TestExecutorAttachesUseWindowsGen2VMToContext(t *testing.T) { + t.Parallel() + g := gomega.NewWithT(t) + + // Gen2-only SKU asking for Gen2 → executor flags the batch. + mockGen2 := &recordingClient{} + execGen2 := newExecutor(mockGen2) + rGen2 := makeReqGen2("m-1", tpl("Standard_D2als_v7", []string{"1"}, nil), true) + execGen2.executeBatch(context.Background(), makeBatch(rGen2)) + callsGen2 := mockGen2.snapshot() + g.Expect(callsGen2).To(gomega.HaveLen(1)) + g.Expect(FakeUseWindowsGen2VMFromContext(callsGen2[0].ctx)).To(gomega.BeTrue(), + "a Gen2 batch should set the UseWindowsGen2VM header") + + // Default (no Gen2 request) → flag absent. + mockDefault := &recordingClient{} + execDefault := newExecutor(mockDefault) + rDefault := makeReq("m-2", tpl("Standard_D2s_v3", []string{"1"}, nil)) + execDefault.executeBatch(context.Background(), makeBatch(rDefault)) + callsDefault := mockDefault.snapshot() + g.Expect(callsDefault).To(gomega.HaveLen(1)) + g.Expect(FakeUseWindowsGen2VMFromContext(callsDefault[0].ctx)).To(gomega.BeFalse(), + "a non-Gen2 batch should not set the UseWindowsGen2VM header") +} + func TestExecutorDistributesErrorToAllCallers(t *testing.T) { t.Parallel() g := gomega.NewWithT(t) @@ -246,7 +279,7 @@ func TestConcurrentRequestsBatchThroughClient(t *testing.T) { for i := 0; i < n; i++ { go func(i int) { defer wg.Done() - _, _ = client.BeginCreateWithBatch(ctx, "rg", "cluster", "pool", fmt.Sprintf("machine-%d", i), tmpl) + _, _ = client.BeginCreateWithBatch(ctx, "rg", "cluster", "pool", fmt.Sprintf("machine-%d", i), tmpl, false) }(i) } wg.Wait() diff --git a/pkg/providers/azclient/aksmachinesheaderbatch/fakeheader.go b/pkg/providers/azclient/aksmachinesheaderbatch/fakeheader.go index cc59767c8e..77d85387e9 100644 --- a/pkg/providers/azclient/aksmachinesheaderbatch/fakeheader.go +++ b/pkg/providers/azclient/aksmachinesheaderbatch/fakeheader.go @@ -44,3 +44,22 @@ func FakeBatchEntriesFromContext(ctx context.Context) []MachineEntry { } return nil } + +type fakeUseWindowsGen2VMKey struct{} + +// WithFakeUseWindowsGen2VM mirrors the per-batch UseWindowsGen2VM request flag into a context so +// fakes/tests can observe it. Like WithFakeBatchEntries, this has NO production significance: the +// real Azure API reads the UseWindowsGen2VM HTTP header (set via policy.WithHTTPHeader, whose +// context key is unexported), not from this context value. +func WithFakeUseWindowsGen2VM(ctx context.Context, useWindowsGen2VM bool) context.Context { + return context.WithValue(ctx, fakeUseWindowsGen2VMKey{}, useWindowsGen2VM) +} + +// FakeUseWindowsGen2VMFromContext retrieves the mirrored UseWindowsGen2VM flag if present. +// Only used by fakes/tests — see WithFakeUseWindowsGen2VM. +func FakeUseWindowsGen2VMFromContext(ctx context.Context) bool { + if v, ok := ctx.Value(fakeUseWindowsGen2VMKey{}).(bool); ok { + return v + } + return false +} diff --git a/pkg/providers/imagefamily/consts.go b/pkg/providers/imagefamily/consts.go index 70d05068af..52252a1253 100644 --- a/pkg/providers/imagefamily/consts.go +++ b/pkg/providers/imagefamily/consts.go @@ -22,7 +22,11 @@ const ( AKSUbuntuResourceGroup = "AKS-Ubuntu" AKSAzureLinuxResourceGroup = "AKS-AzureLinux" + // AKSWindowsResourceGroup is the resource group hosting the AKS Windows shared image gallery. + AKSWindowsResourceGroup = "AKS-Windows" AKSUbuntuGalleryName = "AKSUbuntu" AKSAzureLinuxGalleryName = "AKSAzureLinux" + // AKSWindowsGalleryName is the shared image gallery name for AKS Windows images. + AKSWindowsGalleryName = "AKSWindows" ) diff --git a/pkg/providers/imagefamily/nodeimageversionsclient.go b/pkg/providers/imagefamily/nodeimageversionsclient.go index b9174ddfab..72ed3b3f25 100644 --- a/pkg/providers/imagefamily/nodeimageversionsclient.go +++ b/pkg/providers/imagefamily/nodeimageversionsclient.go @@ -59,7 +59,7 @@ func (l *NodeImageVersionsClient) List(ctx context.Context, location string) ([] // FilteredNodeImages filters on two conditions // 1. The image is the latest version for the given OS and SKU -// 2. the image belongs to a supported gallery(AKS Ubuntu or Azure Linux) +// 2. the image belongs to a supported gallery (AKS Ubuntu, Azure Linux, or AKS Windows) func FilteredNodeImages(nodeImageVersions []*armcontainerservice.NodeImageVersion) []*armcontainerservice.NodeImageVersion { latestImages := make(map[string]*armcontainerservice.NodeImageVersion) @@ -72,7 +72,7 @@ func FilteredNodeImages(nodeImageVersions []*armcontainerservice.NodeImageVersio version := lo.FromPtr(image.Version) // Skip the galleries that Karpenter does not support - if os != AKSUbuntuGalleryName && os != AKSAzureLinuxGalleryName { + if os != AKSUbuntuGalleryName && os != AKSAzureLinuxGalleryName && os != AKSWindowsGalleryName { continue } diff --git a/pkg/providers/imagefamily/resolver.go b/pkg/providers/imagefamily/resolver.go index f33494734c..f791830263 100644 --- a/pkg/providers/imagefamily/resolver.go +++ b/pkg/providers/imagefamily/resolver.go @@ -250,6 +250,9 @@ func GetImageFamily(familyName *string, fipsMode *v1beta1.FIPSMode, kubernetesVe return &AzureLinux3{Options: parameters} } return &AzureLinux{Options: parameters} + case v1beta1.Windows2022ImageFamily, + v1beta1.Windows2025ImageFamily: + return &Windows{Options: parameters, Family: lo.FromPtr(familyName)} case v1beta1.UbuntuImageFamily: fallthrough default: diff --git a/pkg/providers/imagefamily/windows.go b/pkg/providers/imagefamily/windows.go new file mode 100644 index 0000000000..9edd78261c --- /dev/null +++ b/pkg/providers/imagefamily/windows.go @@ -0,0 +1,151 @@ +/* +Portions Copyright (c) Microsoft Corporation. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package imagefamily + +import ( + "context" + "fmt" + + v1 "k8s.io/api/core/v1" + + "github.com/Azure/karpenter-provider-azure/pkg/apis/v1beta1" + "github.com/Azure/karpenter-provider-azure/pkg/providers/imagefamily/bootstrap" + "github.com/Azure/karpenter-provider-azure/pkg/providers/imagefamily/customscriptsbootstrap" + types "github.com/Azure/karpenter-provider-azure/pkg/providers/imagefamily/types" + "github.com/Azure/karpenter-provider-azure/pkg/providers/launchtemplate/parameters" + "github.com/samber/lo" + + karpv1 "sigs.k8s.io/karpenter/pkg/apis/v1" + "sigs.k8s.io/karpenter/pkg/cloudprovider" + "sigs.k8s.io/karpenter/pkg/scheduling" +) + +// Windows image definitions. These match the SKU names returned by the AKS node image +// versions API (e.g. "windows-2022-containerd-gen2") and live in the AKSWindows shared +// image gallery. Windows is amd64-only. +const ( + Windows2022ContainerdGen2ImageDefinition = "windows-2022-containerd-gen2" + Windows2022ContainerdImageDefinition = "windows-2022-containerd" + + Windows2025Gen2ImageDefinition = "windows-2025-gen2" + Windows2025ImageDefinition = "windows-2025" +) + +// Windows is the image family for AKS Windows nodes. It is parameterized by family +// (one of the v1beta1 Windows*ImageFamily values), each of which maps to a Windows OSSKU. +// +// Windows nodes are only supported via the AKS Machine API provision mode, which always +// uses the AKS-managed shared image gallery (SIG). Accordingly, DefaultImages() returns images +// only when useSIG is true; the community image gallery (CIG) is not supported for Windows. +type Windows struct { + Options *parameters.StaticParameters + Family string +} + +func (w Windows) Name() string { + return w.Family +} + +func (w Windows) DefaultImages(useSIG bool, fipsMode *v1beta1.FIPSMode) []types.DefaultImageOutput { + // FIPS is not available for Windows, and Windows images are only published to the + // AKS-managed shared image gallery (not the community image gallery). + if !useSIG || lo.FromPtr(fipsMode) == v1beta1.FIPSModeFIPS { + return []types.DefaultImageOutput{} + } + + // image provider selects these images in order, first compatible match wins, so the + // preferred (gen2) image is listed first. + switch w.Family { + case v1beta1.Windows2025ImageFamily: + return []types.DefaultImageOutput{ + windowsImage(Windows2025Gen2ImageDefinition, v1beta1.HyperVGenerationV2, "aks-windows-2025-gen2"), + windowsImage(Windows2025ImageDefinition, v1beta1.HyperVGenerationV1, "aks-windows-2025"), + } + case v1beta1.Windows2022ImageFamily: + fallthrough + default: + return []types.DefaultImageOutput{ + windowsImage(Windows2022ContainerdGen2ImageDefinition, v1beta1.HyperVGenerationV2, "aks-windows-2022-containerd-gen2"), + windowsImage(Windows2022ContainerdImageDefinition, v1beta1.HyperVGenerationV1, "aks-windows-2022-containerd"), + } + } +} + +// windowsImage builds a Windows DefaultImageOutput for the AKSWindows shared image gallery. +// PublicGalleryURL is intentionally left empty: Windows is SIG-only. +func windowsImage(imageDefinition, hyperVGeneration, distro string) types.DefaultImageOutput { + return types.DefaultImageOutput{ + GalleryResourceGroup: AKSWindowsResourceGroup, + GalleryName: AKSWindowsGalleryName, + ImageDefinition: imageDefinition, + Requirements: scheduling.NewRequirements( + scheduling.NewRequirement(v1.LabelArchStable, v1.NodeSelectorOpIn, karpv1.ArchitectureAmd64), + scheduling.NewRequirement(v1beta1.LabelSKUHyperVGeneration, v1.NodeSelectorOpIn, hyperVGeneration), + ), + Distro: distro, + } +} + +// ScriptlessCustomData is not supported for Windows. Windows nodes are provisioned via the +// AKS Machine API provision mode, which does not use this bootstrap path. +func (w Windows) ScriptlessCustomData( + _ *bootstrap.KubeletConfiguration, + _ []v1.Taint, + _ map[string]string, + _ *string, + _ *cloudprovider.InstanceType, +) bootstrap.Bootstrapper { + return windowsUnsupportedBootstrapper{family: w.Family} +} + +// CustomScriptsNodeBootstrapping is not supported for Windows. Windows nodes are provisioned +// via the AKS Machine API provision mode, which does not use this bootstrap path. +func (w Windows) CustomScriptsNodeBootstrapping( + _ *bootstrap.KubeletConfiguration, + _ []v1.Taint, + _ []v1.Taint, + _ map[string]string, + _ *cloudprovider.InstanceType, + _ string, + _ string, + _ types.NodeBootstrappingAPI, + _ *v1beta1.FIPSMode, + _ *v1beta1.LocalDNS, + _ *v1beta1.ArtifactStreaming, + _ *v1beta1.LinuxOSConfiguration, +) customscriptsbootstrap.Bootstrapper { + return windowsUnsupportedBootstrapper{family: w.Family} +} + +// windowsUnsupportedBootstrapper implements both bootstrap.Bootstrapper and +// customscriptsbootstrap.Bootstrapper, returning a clear error if a non-AKS-Machine-API +// provision mode ever attempts to bootstrap a Windows node. +type windowsUnsupportedBootstrapper struct { + family string +} + +func (b windowsUnsupportedBootstrapper) Script() (string, error) { + return "", b.err() +} + +func (b windowsUnsupportedBootstrapper) GetCustomDataAndCSE(_ context.Context) (string, string, error) { + return "", "", b.err() +} + +func (b windowsUnsupportedBootstrapper) err() error { + return fmt.Errorf("windows image family %q is only supported with PROVISION_MODE=aksmachineapi", b.family) +} diff --git a/pkg/providers/imagefamily/windows_test.go b/pkg/providers/imagefamily/windows_test.go new file mode 100644 index 0000000000..1541b9e736 --- /dev/null +++ b/pkg/providers/imagefamily/windows_test.go @@ -0,0 +1,96 @@ +/* +Portions Copyright (c) Microsoft Corporation. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package imagefamily_test + +import ( + "context" + "testing" + + "github.com/Azure/karpenter-provider-azure/pkg/apis/v1beta1" + "github.com/Azure/karpenter-provider-azure/pkg/providers/imagefamily" + . "github.com/onsi/gomega" + "github.com/samber/lo" +) + +func TestWindows_Name(t *testing.T) { + g := NewWithT(t) + for _, fam := range []string{ + v1beta1.Windows2022ImageFamily, + v1beta1.Windows2025ImageFamily, + } { + w := imagefamily.Windows{Family: fam} + g.Expect(w.Name()).To(Equal(fam)) + } +} + +func TestWindows_DefaultImages(t *testing.T) { + g := NewWithT(t) + + // Windows2022 (SIG) returns gen2 first, then gen1, all in the AKSWindows gallery. + images := imagefamily.Windows{Family: v1beta1.Windows2022ImageFamily}.DefaultImages(true, nil) + g.Expect(images).To(HaveLen(2)) + + g.Expect(images[0].GalleryName).To(Equal(imagefamily.AKSWindowsGalleryName)) + g.Expect(images[0].GalleryResourceGroup).To(Equal(imagefamily.AKSWindowsResourceGroup)) + g.Expect(images[0].ImageDefinition).To(Equal(imagefamily.Windows2022ContainerdGen2ImageDefinition)) + g.Expect(images[0].Distro).To(Equal("aks-windows-2022-containerd-gen2")) + // gen2 image must require HyperV gen2 and amd64 + g.Expect(images[0].Requirements.Get(v1beta1.LabelSKUHyperVGeneration).Has(v1beta1.HyperVGenerationV2)).To(BeTrue()) + + g.Expect(images[1].ImageDefinition).To(Equal(imagefamily.Windows2022ContainerdImageDefinition)) + g.Expect(images[1].Requirements.Get(v1beta1.LabelSKUHyperVGeneration).Has(v1beta1.HyperVGenerationV1)).To(BeTrue()) + + // Windows2025 (SIG) returns gen2 first, then gen1, all in the AKSWindows gallery. + images2025 := imagefamily.Windows{Family: v1beta1.Windows2025ImageFamily}.DefaultImages(true, nil) + g.Expect(images2025).To(HaveLen(2)) + g.Expect(images2025[0].ImageDefinition).To(Equal(imagefamily.Windows2025Gen2ImageDefinition)) + g.Expect(images2025[1].ImageDefinition).To(Equal(imagefamily.Windows2025ImageDefinition)) + + // Windows is SIG-only: CIG (useSIG=false) yields no images. + g.Expect(imagefamily.Windows{Family: v1beta1.Windows2022ImageFamily}.DefaultImages(false, nil)).To(BeEmpty()) + + // FIPS is not available for Windows. + g.Expect(imagefamily.Windows{Family: v1beta1.Windows2022ImageFamily}.DefaultImages(true, lo.ToPtr(v1beta1.FIPSModeFIPS))).To(BeEmpty()) +} + +func TestWindows_GetImageFamily(t *testing.T) { + g := NewWithT(t) + for _, fam := range []string{ + v1beta1.Windows2022ImageFamily, + v1beta1.Windows2025ImageFamily, + } { + resolved := imagefamily.GetImageFamily(lo.ToPtr(fam), nil, "1.30.0", nil) + w, ok := resolved.(*imagefamily.Windows) + g.Expect(ok).To(BeTrue(), "GetImageFamily(%s) should return *Windows", fam) + g.Expect(w.Family).To(Equal(fam)) + } +} + +func TestWindows_BootstrapMethodsUnsupported(t *testing.T) { + g := NewWithT(t) + w := imagefamily.Windows{Family: v1beta1.Windows2022ImageFamily} + + // Both bootstrap paths are only valid for non-Windows; for Windows they must error + // (Windows is provisioned via the AKS Machine API path, not these bootstrappers). + _, err := w.ScriptlessCustomData(nil, nil, nil, nil, nil).Script() + g.Expect(err).To(HaveOccurred()) + g.Expect(err.Error()).To(ContainSubstring("aksmachineapi")) + + _, _, err = w.CustomScriptsNodeBootstrapping(nil, nil, nil, nil, nil, "", "", nil, nil, nil, nil, nil).GetCustomDataAndCSE(context.Background()) + g.Expect(err).To(HaveOccurred()) + g.Expect(err.Error()).To(ContainSubstring("aksmachineapi")) +} diff --git a/pkg/providers/instance/aksmachineinstance.go b/pkg/providers/instance/aksmachineinstance.go index aedd8549fd..789a1ec3d6 100644 --- a/pkg/providers/instance/aksmachineinstance.go +++ b/pkg/providers/instance/aksmachineinstance.go @@ -19,9 +19,11 @@ package instance import ( "context" "fmt" + "net/http" "sync" "time" + "github.com/Azure/azure-sdk-for-go/sdk/azcore/policy" "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v9" "github.com/samber/lo" "k8s.io/apimachinery/pkg/util/sets" @@ -33,6 +35,7 @@ import ( "github.com/Azure/karpenter-provider-azure/pkg/apis/v1beta1" "github.com/Azure/karpenter-provider-azure/pkg/cache" "github.com/Azure/karpenter-provider-azure/pkg/consts" + "github.com/Azure/karpenter-provider-azure/pkg/operator/options" "github.com/Azure/karpenter-provider-azure/pkg/providers/allocationstrategy" "github.com/Azure/karpenter-provider-azure/pkg/providers/azclient" "github.com/Azure/karpenter-provider-azure/pkg/providers/imagefamily" @@ -205,7 +208,15 @@ func (p *DefaultAKSMachineProvider) BeginCreate( nodeClaim *karpv1.NodeClaim, instanceTypes []*corecloudprovider.InstanceType, ) (*AKSMachinePromise, error) { - aksMachineName, err := GetAKSMachineNameFromNodeClaimName(nodeClaim.Name) + isWindows := v1beta1.IsWindowsImageFamily(lo.FromPtr(nodeClass.Spec.ImageFamily)) + var aksMachineName string + var err error + if isWindows { + maxNameLength := WindowsMachineNameMaxLength(options.FromContext(ctx).AKSMachinesPoolName) + aksMachineName, err = GetWindowsAKSMachineName(nodeClaim.Name, maxNameLength) + } else { + aksMachineName, err = GetLinuxAKSMachineName(nodeClaim.Name) + } if err != nil { return nil, fmt.Errorf("failed to generate AKS machine name from NodeClaim name %q: %w", nodeClaim.Name, err) } @@ -448,6 +459,9 @@ func (p *DefaultAKSMachineProvider) beginCreateMachine( return nil, fmt.Errorf("failed to build AKS machine template from template: %w", err) } + // Decide whether to request a Generation 2 Windows image from the RP for the selected SKU. + useWindowsGen2VM := shouldUseWindowsGen2VM(nodeClass, instanceType) + // Call the AKS machine API with the template to create the AKS machine instance if logger := log.FromContext(ctx).V(1); logger.Enabled() { logger.Info("creating AKS machine", "aksMachineName", aksMachineName, "instance-type", instanceType.Name, "aksMachine", BuildJSONFromAKSMachine(aksMachineTemplate)) @@ -455,9 +469,28 @@ func (p *DefaultAKSMachineProvider) beginCreateMachine( // Branch between batch and non-batch creation paths. if p.batchCreationEnabled { - return p.beginCreateMachineBatch(ctx, aksMachineTemplate, aksMachineName, instanceType, capacityType, zone) + return p.beginCreateMachineBatch(ctx, aksMachineTemplate, aksMachineName, instanceType, capacityType, zone, useWindowsGen2VM) } - return p.beginCreateMachineNonBatch(ctx, aksMachineTemplate, aksMachineName, instanceType, capacityType, zone) + return p.beginCreateMachineNonBatch(ctx, aksMachineTemplate, aksMachineName, instanceType, capacityType, zone, useWindowsGen2VM) +} + +// shouldUseWindowsGen2VM reports whether the AKS machine create for this NodeClass and selected +// instance type should request a Generation 2 Windows image from the AKS RP via the +// UseWindowsGen2VM header. +// +// In AKS Machine API mode the RP resolves the Windows image (and its Hyper-V generation) +// server-side from the OSSKU. For the Windows2022/Windows2019 OSSKUs the RP defaults to a +// Generation 1 image and only selects Gen2 when the caller sets UseWindowsGen2VM=true; it then +// rejects the create if the chosen VM size does not support Gen2. Karpenter selects the cheapest +// compatible SKU, which is frequently Gen2-only, so we request Gen2 exactly when the selected SKU +// supports it: Gen2-only and dual-generation SKUs get Gen2 (preferred), while Gen1-only SKUs fall +// back to the RP's Gen1 default. This mirrors the gen2-then-gen1 preference of the in-provider +// (VM-direct) Windows image family and lets Windows NodePools provision on any SKU generation. +func shouldUseWindowsGen2VM(nodeClass *v1beta1.AKSNodeClass, instanceType *corecloudprovider.InstanceType) bool { + if !v1beta1.IsWindowsImageFamily(lo.FromPtr(nodeClass.Spec.ImageFamily)) { + return false + } + return instanceType.Requirements.Get(v1beta1.LabelSKUHyperVGeneration).Has(v1beta1.HyperVGenerationV2) } // beginCreateMachineBatch handles the batch creation path using the AKS machines header batch API and GET-based poller. @@ -468,8 +501,9 @@ func (p *DefaultAKSMachineProvider) beginCreateMachineBatch( instanceType *corecloudprovider.InstanceType, capacityType string, zone string, + useWindowsGen2VM bool, ) (*AKSMachinePromise, error) { - handlableError, err := p.azClient.AKSMachinesBatchClient().BeginCreateWithBatch(ctx, p.clusterResourceGroup, p.clusterName, p.aksMachinesPoolName, aksMachineName, aksMachineTemplate) + handlableError, err := p.azClient.AKSMachinesBatchClient().BeginCreateWithBatch(ctx, p.clusterResourceGroup, p.clusterName, p.aksMachinesPoolName, aksMachineName, aksMachineTemplate, useWindowsGen2VM) if err != nil { return nil, fmt.Errorf("failed to begin create AKS machine %q, unhandled error: %w", aksMachineName, err) } @@ -530,7 +564,12 @@ func (p *DefaultAKSMachineProvider) beginCreateMachineNonBatch( instanceType *corecloudprovider.InstanceType, capacityType string, zone string, + useWindowsGen2VM bool, ) (*AKSMachinePromise, error) { + if useWindowsGen2VM { + // Request a Gen2 Windows image from the RP for this create (see shouldUseWindowsGen2VM). + ctx = policy.WithHTTPHeader(ctx, http.Header{consts.HeaderUseWindowsGen2VM: []string{"true"}}) + } poller, err := p.azClient.AKSMachinesClient().BeginCreateOrUpdate(ctx, p.clusterResourceGroup, p.clusterName, p.aksMachinesPoolName, aksMachineName, *aksMachineTemplate, nil) if err != nil { he := offerings.ErrorToHandlableError(err) diff --git a/pkg/providers/instance/aksmachineinstancehelpers.go b/pkg/providers/instance/aksmachineinstancehelpers.go index 8af65cc312..9d1f2ba52c 100644 --- a/pkg/providers/instance/aksmachineinstancehelpers.go +++ b/pkg/providers/instance/aksmachineinstancehelpers.go @@ -63,13 +63,9 @@ func (p *DefaultAKSMachineProvider) buildAKSMachineTemplate(ctx context.Context, // NodeImageVersion // E.g., "AKSUbuntu-2204gen2containerd-2023.11.15" - vmImageID, err := p.imageResolver.ResolveNodeImageFromNodeClass(nodeClass, instanceType) + nodeImageVersionPtr, err := p.resolveNodeImageVersion(nodeClass, instanceType) if err != nil { - return nil, fmt.Errorf("failed to resolve VM image ID: %w", err) - } - nodeImageVersion, err := utils.GetAKSMachineNodeImageVersionFromImageID(vmImageID) - if err != nil { - return nil, fmt.Errorf("failed to convert VM image ID to NodeImageVersion: %w", err) + return nil, fmt.Errorf("failed to resolve node image version: %w", err) } // GPUProfile @@ -116,7 +112,7 @@ func (p *DefaultAKSMachineProvider) buildAKSMachineTemplate(ctx context.Context, // hashing by design. See batch_field_registry.go for the full field classification. Zones: zones.MakeARMZonesFromAKSLabelZone(zone), Properties: &armcontainerservice.MachineProperties{ - NodeImageVersion: lo.ToPtr(nodeImageVersion), + NodeImageVersion: nodeImageVersionPtr, Network: &armcontainerservice.MachineNetworkProperties{ VnetSubnetID: nodeClass.Spec.VNETSubnetID, // AKS machine API take control, if nil // As of the time of writing, the current version of AKS machine API support just that with nil. That is unlikely to change. @@ -131,21 +127,16 @@ func (p *DefaultAKSMachineProvider) buildAKSMachineTemplate(ctx context.Context, GpuProfile: gpuProfile, }, OperatingSystem: &armcontainerservice.MachineOSProfile{ - OSType: lo.ToPtr(armcontainerservice.OSTypeLinux), + OSType: configureOSType(nodeClass), OSSKU: osSku, OSDiskSizeGB: nodeClass.Spec.OSDiskSizeGB, // AKS machine API defaults it if nil OSDiskType: osDiskType, EnableFIPS: enableFIPS, - LinuxProfile: func() *armcontainerservice.MachineOSProfileLinuxProfile { - linuxOSConfig := configureLinuxOSConfig(nodeClass) - if linuxOSConfig == nil { - return nil - } - return &armcontainerservice.MachineOSProfileLinuxProfile{ - LinuxOSConfig: linuxOSConfig, - } - }(), - // WindowsProfile: nil, + LinuxProfile: configureLinuxProfile(nodeClass), + // WindowsProfile is optional. Windows admin credentials are sourced + // server-side by the AKS RP from the ManagedCluster's WindowsProfile, so + // Karpenter does not need to populate it. TODO(Windows): expose advanced + // AgentPoolWindowsProfile settings (e.g. DisableOutboundNat) if needed. }, Kubernetes: &armcontainerservice.MachineKubernetesProfile{ @@ -284,38 +275,97 @@ func configurePriority(capacityType string) *armcontainerservice.ScaleSetPriorit } } +// imageFamilyToARMOSSKU maps NodeClass image families with a 1:1 OSSKU to the AKS Machine +// API OSSKU. The generic "Ubuntu" family (and any unknown family) is resolved separately by +// defaultUbuntuOSSKU, since it depends on FIPS and the Kubernetes version. +var imageFamilyToARMOSSKU = map[string]armcontainerservice.OSSKU{ + v1beta1.Ubuntu2204ImageFamily: armcontainerservice.OSSKUUbuntu2204, + v1beta1.Ubuntu2404ImageFamily: armcontainerservice.OSSKUUbuntu2404, + v1beta1.AzureLinuxImageFamily: armcontainerservice.OSSKUAzureLinux, + v1beta1.Windows2022ImageFamily: armcontainerservice.OSSKUWindows2022, + v1beta1.Windows2025ImageFamily: armcontainerservice.OSSKUWindows2025, +} + func configureOSSKUAndFIPs(nodeClass *v1beta1.AKSNodeClass, orchestratorVersion string) (*armcontainerservice.OSSKU, *bool, error) { // Counterpart for ProvisionModeBootstrappingClient is in customscriptsbootstrap/provisionclientbootstrap.go if nodeClass.Spec.ImageFamily == nil { return nil, nil, fmt.Errorf("ImageFamily is not set in NodeClass %q", nodeClass.Name) } + family := *nodeClass.Spec.ImageFamily - var ossku armcontainerservice.OSSKU - enableFIPS := lo.FromPtr(nodeClass.Spec.FIPSMode) == v1beta1.FIPSModeFIPS + // FIPS is not applicable to Windows; CEL validation rejects FIPS with a Windows family, + // but guard here as well so we never send EnableFIPS=true for a Windows machine. + enableFIPS := !v1beta1.IsWindowsImageFamily(family) && lo.FromPtr(nodeClass.Spec.FIPSMode) == v1beta1.FIPSModeFIPS - switch *nodeClass.Spec.ImageFamily { - case v1beta1.Ubuntu2204ImageFamily: - ossku = armcontainerservice.OSSKUUbuntu2204 - case v1beta1.Ubuntu2404ImageFamily: - ossku = armcontainerservice.OSSKUUbuntu2404 - case v1beta1.AzureLinuxImageFamily: - ossku = armcontainerservice.OSSKUAzureLinux - case v1beta1.UbuntuImageFamily: - fallthrough - default: - if enableFIPS { - ossku = armcontainerservice.OSSKUUbuntu - } else if imagefamily.UseUbuntu2404(orchestratorVersion) { - ossku = armcontainerservice.OSSKUUbuntu2404 - } else { - ossku = armcontainerservice.OSSKUUbuntu2204 - } + ossku, ok := imageFamilyToARMOSSKU[family] + if !ok { + // Generic "Ubuntu" family or unknown family. + ossku = defaultUbuntuOSSKU(enableFIPS, orchestratorVersion) } return lo.ToPtr(ossku), lo.ToPtr(enableFIPS), nil } +// defaultUbuntuOSSKU resolves the Ubuntu OSSKU for the generic "Ubuntu" image family, +// based on FIPS mode and the Kubernetes version. +func defaultUbuntuOSSKU(enableFIPS bool, orchestratorVersion string) armcontainerservice.OSSKU { + switch { + case enableFIPS: + return armcontainerservice.OSSKUUbuntu + case imagefamily.UseUbuntu2404(orchestratorVersion): + return armcontainerservice.OSSKUUbuntu2404 + default: + return armcontainerservice.OSSKUUbuntu2204 + } +} + +// configureOSType returns the AKS Machine OSType (Linux or Windows) for the NodeClass's +// image family. Windows families map to OSTypeWindows; everything else to OSTypeLinux. +func configureOSType(nodeClass *v1beta1.AKSNodeClass) *armcontainerservice.OSType { + if v1beta1.IsWindowsImageFamily(lo.FromPtr(nodeClass.Spec.ImageFamily)) { + return lo.ToPtr(armcontainerservice.OSTypeWindows) + } + return lo.ToPtr(armcontainerservice.OSTypeLinux) +} + +// resolveNodeImageVersion resolves the AKS Machine API NodeImageVersion for the NodeClass. +// +// For Windows it returns nil: the AKS Machine API's node image version parser splits on "-" +// and expects exactly "{gallery}-{name}-{version}", but Windows image names themselves +// contain hyphens (e.g. "AKSWindows-2022-containerd-gen2-"), so passing an explicit +// version is rejected as invalid. Leaving it nil lets the AKS RP resolve the latest image +// from the OSSKU (it already owns Windows image/credential resolution server-side). +func (p *DefaultAKSMachineProvider) resolveNodeImageVersion(nodeClass *v1beta1.AKSNodeClass, instanceType *corecloudprovider.InstanceType) (*string, error) { + if v1beta1.IsWindowsImageFamily(lo.FromPtr(nodeClass.Spec.ImageFamily)) { + return nil, nil + } + vmImageID, err := p.imageResolver.ResolveNodeImageFromNodeClass(nodeClass, instanceType) + if err != nil { + return nil, fmt.Errorf("failed to resolve VM image ID: %w", err) + } + nodeImageVersion, err := utils.GetAKSMachineNodeImageVersionFromImageID(vmImageID) + if err != nil { + return nil, fmt.Errorf("failed to convert VM image ID to NodeImageVersion: %w", err) + } + return lo.ToPtr(nodeImageVersion), nil +} + +// configureLinuxProfile builds the Machine LinuxProfile. It returns nil for Windows nodes +// (not applicable) and when no Linux OS config is set. +func configureLinuxProfile(nodeClass *v1beta1.AKSNodeClass) *armcontainerservice.MachineOSProfileLinuxProfile { + if v1beta1.IsWindowsImageFamily(lo.FromPtr(nodeClass.Spec.ImageFamily)) { + return nil + } + linuxOSConfig := configureLinuxOSConfig(nodeClass) + if linuxOSConfig == nil { + return nil + } + return &armcontainerservice.MachineOSProfileLinuxProfile{ + LinuxOSConfig: linuxOSConfig, + } +} + func configureTaints(nodeClaim *karpv1.NodeClaim) ([]*string, []*string) { generalTaints, startupTaints := utils.ExtractTaints(nodeClaim) allTaints := lo.Flatten([][]v1.Taint{generalTaints, startupTaints}) diff --git a/pkg/providers/instance/aksmachineinstancehelpers_test.go b/pkg/providers/instance/aksmachineinstancehelpers_test.go index 498a9bd83d..e8c36e9e78 100644 --- a/pkg/providers/instance/aksmachineinstancehelpers_test.go +++ b/pkg/providers/instance/aksmachineinstancehelpers_test.go @@ -174,6 +174,33 @@ var _ = Describe("AKSMachineInstance Helper Functions", func() { }) }) + Context("Windows Image Families", func() { + It("should configure Windows2022", func() { + nodeClass.Spec.ImageFamily = lo.ToPtr(v1beta1.Windows2022ImageFamily) + ossku, enableFIPs, err := configureOSSKUAndFIPs(nodeClass, "1.30.0") + Expect(err).ToNot(HaveOccurred()) + Expect(*ossku).To(Equal(armcontainerservice.OSSKUWindows2022)) + Expect(*enableFIPs).To(BeFalse()) + }) + + It("should configure Windows2025", func() { + nodeClass.Spec.ImageFamily = lo.ToPtr(v1beta1.Windows2025ImageFamily) + ossku, enableFIPs, err := configureOSSKUAndFIPs(nodeClass, "1.30.0") + Expect(err).ToNot(HaveOccurred()) + Expect(*ossku).To(Equal(armcontainerservice.OSSKUWindows2025)) + Expect(*enableFIPs).To(BeFalse()) + }) + + It("should force EnableFIPS false for Windows even if FIPSMode is FIPS", func() { + nodeClass.Spec.ImageFamily = lo.ToPtr(v1beta1.Windows2022ImageFamily) + nodeClass.Spec.FIPSMode = lo.ToPtr(v1beta1.FIPSModeFIPS) + ossku, enableFIPs, err := configureOSSKUAndFIPs(nodeClass, "1.30.0") + Expect(err).ToNot(HaveOccurred()) + Expect(*ossku).To(Equal(armcontainerservice.OSSKUWindows2022)) + Expect(*enableFIPs).To(BeFalse()) + }) + }) + Context("Error Cases", func() { It("should return error when ImageFamily is nil", func() { nodeClass.Spec.ImageFamily = nil @@ -198,6 +225,30 @@ var _ = Describe("AKSMachineInstance Helper Functions", func() { }) }) + Context("configureOSType", func() { + It("should return Linux for Linux image families", func() { + for _, fam := range []string{ + v1beta1.UbuntuImageFamily, + v1beta1.Ubuntu2204ImageFamily, + v1beta1.Ubuntu2404ImageFamily, + v1beta1.AzureLinuxImageFamily, + } { + nodeClass.Spec.ImageFamily = lo.ToPtr(fam) + Expect(*configureOSType(nodeClass)).To(Equal(armcontainerservice.OSTypeLinux), "family %s", fam) + } + }) + + It("should return Windows for Windows image families", func() { + for _, fam := range []string{ + v1beta1.Windows2022ImageFamily, + v1beta1.Windows2025ImageFamily, + } { + nodeClass.Spec.ImageFamily = lo.ToPtr(fam) + Expect(*configureOSType(nodeClass)).To(Equal(armcontainerservice.OSTypeWindows), "family %s", fam) + } + }) + }) + // Currently, we will use "nodeInitializationTaints" field for all taints, as "taints" field are subjected to server-side reconciliation and extra validation // Server-side reconciliation is not necessarily a bad thing, but needs to resolve validation conflicts at least. E.g., system node cannot have hard taints other than CriticalAddonsOnly, per AKS Machine API. Context("configureTaints", func() { @@ -945,4 +996,43 @@ var _ = Describe("AKSMachineInstance Helper Functions", func() { Expect(*profile.Driver).To(Equal(armcontainerservice.GPUDriverNone)) }) }) + + Context("shouldUseWindowsGen2VM", func() { + // withHyperVGenerations returns instanceType with the given supported Hyper-V generation + // label values (e.g. "1", "2"), matching how instance types are labeled in production. + withHyperVGenerations := func(gens ...string) *corecloudprovider.InstanceType { + reqs := scheduling.NewRequirements( + scheduling.NewRequirement(v1.LabelArchStable, v1.NodeSelectorOpIn, karpv1.ArchitectureAmd64), + ) + if len(gens) > 0 { + reqs.Add(scheduling.NewRequirement(v1beta1.LabelSKUHyperVGeneration, v1.NodeSelectorOpIn, gens...)) + } + return &corecloudprovider.InstanceType{Name: "Standard_Test", Requirements: reqs} + } + + It("should request Gen2 for a Windows family on a Gen2-only SKU", func() { + nodeClass.Spec.ImageFamily = lo.ToPtr(v1beta1.Windows2022ImageFamily) + Expect(shouldUseWindowsGen2VM(nodeClass, withHyperVGenerations(v1beta1.HyperVGenerationV2))).To(BeTrue()) + }) + + It("should request Gen2 for a Windows family on a dual-generation SKU (Gen2 preferred)", func() { + nodeClass.Spec.ImageFamily = lo.ToPtr(v1beta1.Windows2025ImageFamily) + Expect(shouldUseWindowsGen2VM(nodeClass, withHyperVGenerations(v1beta1.HyperVGenerationV1, v1beta1.HyperVGenerationV2))).To(BeTrue()) + }) + + It("should NOT request Gen2 for a Windows family on a Gen1-only SKU", func() { + nodeClass.Spec.ImageFamily = lo.ToPtr(v1beta1.Windows2022ImageFamily) + Expect(shouldUseWindowsGen2VM(nodeClass, withHyperVGenerations(v1beta1.HyperVGenerationV1))).To(BeFalse()) + }) + + It("should be case-insensitive about the Windows image family", func() { + nodeClass.Spec.ImageFamily = lo.ToPtr(strings.ToLower(v1beta1.Windows2022ImageFamily)) + Expect(shouldUseWindowsGen2VM(nodeClass, withHyperVGenerations(v1beta1.HyperVGenerationV2))).To(BeTrue()) + }) + + It("should NOT request Gen2 for a non-Windows family even on a Gen2-capable SKU", func() { + nodeClass.Spec.ImageFamily = lo.ToPtr(v1beta1.Ubuntu2204ImageFamily) + Expect(shouldUseWindowsGen2VM(nodeClass, withHyperVGenerations(v1beta1.HyperVGenerationV2))).To(BeFalse()) + }) + }) }) diff --git a/pkg/providers/instance/aksmachineinstanceutils.go b/pkg/providers/instance/aksmachineinstanceutils.go index 3b3a0eb212..8813097ed9 100644 --- a/pkg/providers/instance/aksmachineinstanceutils.go +++ b/pkg/providers/instance/aksmachineinstanceutils.go @@ -181,9 +181,57 @@ func FindNodePoolFromAKSMachine(ctx context.Context, aksMachine *armcontainerser // ASSUMPTION: NodeClaim name is in the format of - // If total length exceeds AKS machine name limit, the exceeded part will be replaced with another deterministic hash. // E.g., "thisisalongnodepoolname-a1b2c" --> "thisisalongnoz9y8x7-a1b2c" -func GetAKSMachineNameFromNodeClaimName(nodeClaimName string) (string, error) { - const maxAKSMachineNameLength = 34 // Defined by AKS machine API. - const prefixHashLength = 6 // The length of the hashed part replacing the exceeded part of the prefix. +const ( + // maxAKSMachineNameLength is the maximum AKS machine name length defined by the AKS machine API. + maxAKSMachineNameLength = 34 + + // ReservedNAPMachinesPoolName is the agent pool name reserved for managed Node Auto-Provisioning + // (NAP). The AKS RP grants this pool a larger Windows machine-name budget than custom pools. + ReservedNAPMachinesPoolName = "aksmanagedap" + + // Windows AKS machine names are ultimately bounded by the Windows NetBIOS computer-name limit + // (15 chars) once the AKS RP composes the VM name from the pool and machine names. The usable + // machine-name budget therefore depends on the agent pool, and these values mirror the AKS RP + // machine-name validation: + // - reserved NAP pool (aksmanagedap): VM name = "aks" + => machine <= 12 + // - any custom/self-hosted pool: VM name = "aks" + + "-" + => machine <= 5 + windowsMachineNameMaxLenNAP = 12 + windowsMachineNameMaxLenCustom = 5 +) + +// WindowsMachineNameMaxLength returns the maximum usable Windows AKS machine-name length for the given +// machines agent pool, mirroring the AKS RP validation (12 in the reserved NAP pool, 5 in custom pools). +// Custom (self-hosted) pools have a much smaller budget because the RP-composed VM name also includes +// the pool name; this is what allows self-hosted Windows provisioning into a short-named machines pool. +func WindowsMachineNameMaxLength(machinesPoolName string) int { + if machinesPoolName == ReservedNAPMachinesPoolName { + return windowsMachineNameMaxLenNAP + } + return windowsMachineNameMaxLenCustom +} + +// GetWindowsAKSMachineName derives a short, deterministic, hyphen-free AKS machine name from a NodeClaim +// name for Windows, bounded to maxNameLength (see WindowsMachineNameMaxLength). Windows machine names are +// constrained by the NetBIOS computer-name limit, so the (much longer) NodeClaim name is hashed rather +// than truncated. Format: "w" + (maxNameLength-1)-char alphanumeric hash (starts with a letter; no +// hyphens). Collision-resistance scales with maxNameLength: the reserved NAP budget of 12 gives an +// 11-char (~36^11 ≈ 2^57) hash space, far exceeding the NodeClaim's input entropy at any cluster scale. +func GetWindowsAKSMachineName(nodeClaimName string, maxNameLength int) (string, error) { + if maxNameLength < 2 { + return "", fmt.Errorf("windows AKS machine name max length must be >= 2, got %d", maxNameLength) + } + hash, err := utils.GetAlphanumericHash(nodeClaimName, maxNameLength-1) + if err != nil { + return "", fmt.Errorf("failed to hash NodeClaim name %q for Windows AKS machine name: %w", nodeClaimName, err) + } + return "w" + hash, nil +} + +// GetLinuxAKSMachineName derives a valid AKS machine name (<= 34 chars) from a NodeClaim name for Linux. +// Names within the limit are used verbatim; longer names keep the legible prefix and trailing suffix and +// hash the overflowing middle so the result stays deterministic and collision-resistant. +func GetLinuxAKSMachineName(nodeClaimName string) (string, error) { + const prefixHashLength = 6 // The length of the hashed part replacing the exceeded part of the prefix. // If 6, given alphanumeric hash, there will be a total of 36^6 = 2,176,782,336 combinations. if len(nodeClaimName) <= maxAKSMachineNameLength { diff --git a/pkg/providers/instance/aksmachineinstanceutils_test.go b/pkg/providers/instance/aksmachineinstanceutils_test.go index 213beb81fd..f74c772e91 100644 --- a/pkg/providers/instance/aksmachineinstanceutils_test.go +++ b/pkg/providers/instance/aksmachineinstanceutils_test.go @@ -239,10 +239,10 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { }) }) - Context("GetAKSMachineNameFromNodeClaimName", func() { + Context("GetLinuxAKSMachineName / GetWindowsAKSMachineName", func() { It("should return the same name when under length limit", func() { nodeClaimName := "default-a1b2c" - machineName, err := GetAKSMachineNameFromNodeClaimName(nodeClaimName) + machineName, err := GetLinuxAKSMachineName(nodeClaimName) Expect(err).ToNot(HaveOccurred()) Expect(machineName).To(Equal(nodeClaimName)) @@ -250,7 +250,7 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { It("should handle short names correctly", func() { nodeClaimName := "d-a1b2c" - machineName, err := GetAKSMachineNameFromNodeClaimName(nodeClaimName) + machineName, err := GetLinuxAKSMachineName(nodeClaimName) Expect(err).ToNot(HaveOccurred()) Expect(machineName).To(Equal(nodeClaimName)) @@ -258,7 +258,7 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { It("should return the same name when at the length limit", func() { nodeClaimName := "123456789-123456789-12345678-a1b2c" - machineName, err := GetAKSMachineNameFromNodeClaimName(nodeClaimName) + machineName, err := GetLinuxAKSMachineName(nodeClaimName) Expect(err).ToNot(HaveOccurred()) Expect(machineName).To(Equal(nodeClaimName)) @@ -266,7 +266,7 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { It("should truncate and hash when at the length limit +1", func() { nodeClaimName := "123456789-123456789-123456789-a1b2c" - machineName, err := GetAKSMachineNameFromNodeClaimName(nodeClaimName) + machineName, err := GetLinuxAKSMachineName(nodeClaimName) Expect(err).ToNot(HaveOccurred()) Expect(machineName).To(HaveLen(34)) @@ -277,8 +277,8 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { It("should truncate and hash differently when at the length limit +1", func() { nodeClaimName1 := "123456789-123456789-123456789-a1b2c" nodeClaimName2 := "123456789-123456789-12345678X-a1b2c" - machineName1, err1 := GetAKSMachineNameFromNodeClaimName(nodeClaimName1) - machineName2, err2 := GetAKSMachineNameFromNodeClaimName(nodeClaimName2) + machineName1, err1 := GetLinuxAKSMachineName(nodeClaimName1) + machineName2, err2 := GetLinuxAKSMachineName(nodeClaimName2) Expect(err1).ToNot(HaveOccurred()) Expect(err2).ToNot(HaveOccurred()) @@ -287,7 +287,7 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { It("should truncate and hash when above the length limit", func() { nodeClaimName := "123456789-123456789-123456789-123456789-a1b2c" - machineName, err := GetAKSMachineNameFromNodeClaimName(nodeClaimName) + machineName, err := GetLinuxAKSMachineName(nodeClaimName) Expect(err).ToNot(HaveOccurred()) Expect(machineName).To(HaveLen(34)) @@ -297,8 +297,8 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { It("should produce deterministic results for same input", func() { nodeClaimName := "consistent-very-long-nodepool-name-test-xyz12" - machineName1, err1 := GetAKSMachineNameFromNodeClaimName(nodeClaimName) - machineName2, err2 := GetAKSMachineNameFromNodeClaimName(nodeClaimName) + machineName1, err1 := GetLinuxAKSMachineName(nodeClaimName) + machineName2, err2 := GetLinuxAKSMachineName(nodeClaimName) Expect(err1).ToNot(HaveOccurred()) Expect(err2).ToNot(HaveOccurred()) @@ -307,7 +307,7 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { It("should preserve the suffix from NodeClaim name", func() { longNodeClaimName := "extremely-long-nodepool-name-that-definitely-exceeds-limits-xyz7890" - machineName, err := GetAKSMachineNameFromNodeClaimName(longNodeClaimName) + machineName, err := GetLinuxAKSMachineName(longNodeClaimName) Expect(err).ToNot(HaveOccurred()) Expect(machineName).To(HaveSuffix("-xyz7890")) @@ -320,12 +320,12 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { nodeClaimName4 := "-a-a-a-a-a-a-a-a-a-a-a-a-a--a1b2c" nodeClaimName5 := "-----------------------------a1b2c" nodeClaimName6 := "-------------------------------a1b2c" - machineName1, err1 := GetAKSMachineNameFromNodeClaimName(nodeClaimName1) - machineName2, err2 := GetAKSMachineNameFromNodeClaimName(nodeClaimName2) - machineName3, err3 := GetAKSMachineNameFromNodeClaimName(nodeClaimName3) - machineName4, err4 := GetAKSMachineNameFromNodeClaimName(nodeClaimName4) - machineName5, err5 := GetAKSMachineNameFromNodeClaimName(nodeClaimName5) - machineName6, err6 := GetAKSMachineNameFromNodeClaimName(nodeClaimName6) + machineName1, err1 := GetLinuxAKSMachineName(nodeClaimName1) + machineName2, err2 := GetLinuxAKSMachineName(nodeClaimName2) + machineName3, err3 := GetLinuxAKSMachineName(nodeClaimName3) + machineName4, err4 := GetLinuxAKSMachineName(nodeClaimName4) + machineName5, err5 := GetLinuxAKSMachineName(nodeClaimName5) + machineName6, err6 := GetLinuxAKSMachineName(nodeClaimName6) Expect(err1).ToNot(HaveOccurred()) Expect(err2).ToNot(HaveOccurred()) @@ -350,6 +350,60 @@ var _ = Describe("AKSMachineInstanceUtils Helper Functions", func() { Expect(machineName6).To(HaveSuffix("-a1b2c")) Expect(machineName6).To(HavePrefix("----------------------")) }) + + It("should produce a short (<=12 char) hyphen-free name for Windows", func() { + // Windows AKS machine names are limited to 12 chars (NetBIOS computer-name limit). + for _, nodeClaimName := range []string{ + "default-a1b2c", + "extremely-long-nodepool-name-that-definitely-exceeds-limits-xyz7890", + } { + machineName, err := GetWindowsAKSMachineName(nodeClaimName, 12) + Expect(err).ToNot(HaveOccurred()) + Expect(len(machineName)).To(BeNumerically("<=", 12), "name %q too long", machineName) + Expect(machineName).ToNot(ContainSubstring("-"), "Windows machine name must not contain hyphens") + // Must start with a letter (valid Windows computer name). + Expect(machineName[0]).To(BeNumerically(">=", byte('a'))) + Expect(machineName[0]).To(BeNumerically("<=", byte('z'))) + } + }) + + It("should be deterministic and unique for Windows", func() { + n1, err1 := GetWindowsAKSMachineName("nodepool-windows-a1b2c", 12) + n1Again, _ := GetWindowsAKSMachineName("nodepool-windows-a1b2c", 12) + n2, err2 := GetWindowsAKSMachineName("nodepool-windows-x9y8z", 12) + Expect(err1).ToNot(HaveOccurred()) + Expect(err2).ToNot(HaveOccurred()) + Expect(n1).To(Equal(n1Again), "must be deterministic") + Expect(n1).ToNot(Equal(n2), "distinct NodeClaims must map to distinct machine names") + }) + + It("should honor the pool-derived max length for Windows", func() { + // Custom/self-hosted pool budget is 5 chars ("w" + 4-char hash). + short, err := GetWindowsAKSMachineName("nodepool-windows-a1b2c", WindowsMachineNameMaxLength("winmp")) + Expect(err).ToNot(HaveOccurred()) + Expect(len(short)).To(BeNumerically("<=", 5)) + Expect(short).To(HavePrefix("w")) + Expect(short).ToNot(ContainSubstring("-")) + + // Reserved NAP pool budget is 12 chars. + long, err := GetWindowsAKSMachineName("nodepool-windows-a1b2c", WindowsMachineNameMaxLength("aksmanagedap")) + Expect(err).ToNot(HaveOccurred()) + Expect(len(long)).To(BeNumerically("<=", 12)) + Expect(len(long)).To(BeNumerically(">", 5)) + }) + + It("should reject an unusable Windows max length", func() { + _, err := GetWindowsAKSMachineName("nodepool-windows-a1b2c", 1) + Expect(err).To(HaveOccurred()) + }) + }) + + Context("WindowsMachineNameMaxLength", func() { + It("returns 12 for the reserved NAP pool and 5 for custom pools", func() { + Expect(WindowsMachineNameMaxLength("aksmanagedap")).To(Equal(12)) + Expect(WindowsMachineNameMaxLength("winmp")).To(Equal(5)) + Expect(WindowsMachineNameMaxLength("")).To(Equal(5)) + }) }) Context("GetAKSMachineNameFromNodeClaim", func() { diff --git a/pkg/providers/instancetype/instancetype.go b/pkg/providers/instancetype/instancetype.go index 09b150af90..b38a15bd3d 100644 --- a/pkg/providers/instancetype/instancetype.go +++ b/pkg/providers/instancetype/instancetype.go @@ -152,7 +152,7 @@ func computeRequirements( // Well Known Upstream scheduling.NewRequirement(corev1.LabelInstanceTypeStable, corev1.NodeSelectorOpIn, sku.GetName()), scheduling.NewRequirement(corev1.LabelArchStable, corev1.NodeSelectorOpIn, getArchitecture(architecture)), - scheduling.NewRequirement(corev1.LabelOSStable, corev1.NodeSelectorOpIn, string(corev1.Linux)), + scheduling.NewRequirement(corev1.LabelOSStable, corev1.NodeSelectorOpIn, v1beta1.GetOSForImageFamily(params.ImageFamily)), scheduling.NewRequirement(corev1.LabelTopologyZone, corev1.NodeSelectorOpIn, lo.Map(offerings.Available(), func(o *cloudprovider.Offering, _ int) string { return o.Requirements.Get(corev1.LabelTopologyZone).Any() })...), diff --git a/pkg/utils/image.go b/pkg/utils/image.go index fb654b0dd8..4a1b2c62bc 100644 --- a/pkg/utils/image.go +++ b/pkg/utils/image.go @@ -26,6 +26,10 @@ var ( sigImageIDRegex = regexp.MustCompile(`(?i)/subscriptions/(\S+)/resourceGroups/(\S+)/providers/Microsoft.Compute/galleries/(\S+)/images/(\S+)/versions/(\S+)`) ) +// windowsImageDefinitionPrefix is the prefix AKS Windows SIG image definitions carry +// (e.g. "windows-2022-containerd-gen2"). It is dropped when building the NodeImageVersion. +const windowsImageDefinitionPrefix = "windows-" + // WARNING: not supporting CIG images yet. func GetAKSMachineNodeImageVersionFromImageID(imageID string) (string, error) { if strings.HasPrefix(imageID, "/CommunityGalleries") { @@ -38,6 +42,10 @@ func GetAKSMachineNodeImageVersionFromImageID(imageID string) (string, error) { // Convert from "/subscriptions/10945678-1234-1234-1234-123456789012/resourceGroups/AKS-Ubuntu/providers/Microsoft.Compute/galleries/AKSUbuntu/images/2204gen2containerd/versions/2022.10.03" // to "AKSUbuntu-2204gen2containerd-2022.10.03". +// +// For Windows the image definition is named like "windows-2022-containerd-gen2"; the +// redundant "windows-" prefix is dropped so the result matches the AKS Windows +// NodeImageVersion form, e.g. "AKSWindows-2022-containerd-gen2-20348.4529.251212". func GetAKSMachineNodeImageVersionFromSIGImageID(imageID string) (string, error) { matches := sigImageIDRegex.FindStringSubmatch(imageID) if matches == nil { @@ -52,9 +60,15 @@ func GetAKSMachineNodeImageVersionFromSIGImageID(imageID string) (string, error) prefix := gallery osVersion := definition - // if strings.Contains(prefix, windowsPrefix) { // TODO(Windows) - // osVersion = extractOsVersionForWindows(definition) - // } + if strings.HasPrefix(definition, windowsImageDefinitionPrefix) { + osVersion = extractOSVersionForWindows(definition) + } return strings.Join([]string{prefix, osVersion, version}, "-"), nil } + +// extractOSVersionForWindows drops the leading "windows-" prefix from a Windows SIG image +// definition (e.g. "windows-2022-containerd-gen2" -> "2022-containerd-gen2"). +func extractOSVersionForWindows(definition string) string { + return strings.TrimPrefix(definition, windowsImageDefinitionPrefix) +} diff --git a/pkg/utils/image_test.go b/pkg/utils/image_test.go index 245a65fcd7..1714a48470 100644 --- a/pkg/utils/image_test.go +++ b/pkg/utils/image_test.go @@ -42,6 +42,12 @@ func TestGetAKSMachineNodeImageVersionFromImageID(t *testing.T) { expectedError: "", expectedResult: "AKSUbuntu-2204gen2containerd-2022.10.03", }, + { + name: "Valid Windows SIG image ID drops windows- prefix", + imageID: "/subscriptions/10945678-1234-1234-1234-123456789012/resourceGroups/AKS-Windows/providers/Microsoft.Compute/galleries/AKSWindows/images/windows-2022-containerd-gen2/versions/20348.4529.251212", + expectedError: "", + expectedResult: "AKSWindows-2022-containerd-gen2-20348.4529.251212", + }, { name: "Invalid SIG image ID", imageID: "/subscriptions/invalid/format", @@ -96,6 +102,18 @@ func TestGetAKSMachineNodeImageVersionFromSIGImageID(t *testing.T) { expectedError: "", expectedResult: "AKSUbuntu-2204gen2containerd-2022.10.03-build.1", }, + { + name: "Valid Windows SIG image ID (gen2) drops windows- prefix", + imageID: "/subscriptions/10945678-1234-1234-1234-123456789012/resourceGroups/AKS-Windows/providers/Microsoft.Compute/galleries/AKSWindows/images/windows-2022-containerd-gen2/versions/20348.4529.251212", + expectedError: "", + expectedResult: "AKSWindows-2022-containerd-gen2-20348.4529.251212", + }, + { + name: "Valid Windows Annual SIG image ID drops windows- prefix", + imageID: "/subscriptions/10945678-1234-1234-1234-123456789012/resourceGroups/AKS-Windows/providers/Microsoft.Compute/galleries/AKSWindows/images/windows-23H2-gen2/versions/25398.2025.251212", + expectedError: "", + expectedResult: "AKSWindows-23H2-gen2-25398.2025.251212", + }, { name: "Invalid SIG image ID - missing subscription", imageID: "/resourceGroups/AKS-Ubuntu/providers/Microsoft.Compute/galleries/AKSUbuntu/images/2204gen2containerd/versions/2022.10.03", diff --git a/pkg/utils/utils.go b/pkg/utils/utils.go index 4ede5fe155..913840750a 100644 --- a/pkg/utils/utils.go +++ b/pkg/utils/utils.go @@ -210,7 +210,8 @@ func HasChanged(existing, new any, options *hashstructure.HashOptions) bool { // Be mindful of collision risks with short lengths. Also note that length > 13 provides no additional // collision resistance because the underlying hashstructure library returns a 64-bit hash, which only // fills ~13 base36 characters; extra characters are just leading zeros. -// At the time of writing, this is being used in AKS machine instance provider/GetAKSMachineNameFromNodeClaimName(). See that for context. +// At the time of writing, this is used by the AKS machine instance provider's GetLinuxAKSMachineName / +// GetWindowsAKSMachineName (see pkg/providers/instance/aksmachineinstanceutils.go) for context. func GetAlphanumericHash(input string, length int) (string, error) { if length <= 0 { return "", fmt.Errorf("length must be positive, got %d", length) diff --git a/test/pkg/environment/azure/environment.go b/test/pkg/environment/azure/environment.go index 7a1b3476c1..a6fabab90d 100644 --- a/test/pkg/environment/azure/environment.go +++ b/test/pkg/environment/azure/environment.go @@ -185,7 +185,9 @@ func NewEnvironment(t *testing.T) *Environment { // Default to reserved managed machine agentpool name for NAP azureEnv.MachineAgentPoolName = "aksmanagedap" if azureEnv.InClusterController { - azureEnv.MachineAgentPoolName = "testmpool" + // Self-hosted machines pool name; matches AKS_MACHINES_POOL_NAME used at deploy time. + // Note: Windows machines require an agent pool name <= 6 characters, so keep this short. + azureEnv.MachineAgentPoolName = lo.Ternary(os.Getenv("AKS_MACHINES_POOL_NAME") == "", "testmpool", os.Getenv("AKS_MACHINES_POOL_NAME")) } // Confirm we have a machine pool if azureEnv.InClusterController && azureEnv.IsAKSMachineAPIMode() { @@ -256,6 +258,28 @@ func (env *Environment) AZLinuxNodeClass() *v1beta1.AKSNodeClass { return nodeClass } +// WindowsNodeClass returns an AKSNodeClass configured for the Windows2022 image family. +// Windows nodes are only provisionable in the AKS Machine API provision mode. +func (env *Environment) WindowsNodeClass() *v1beta1.AKSNodeClass { + nodeClass := env.DefaultAKSNodeClass() + nodeClass.Spec.ImageFamily = lo.ToPtr(v1beta1.Windows2022ImageFamily) + return nodeClass +} + +// WindowsNodePool returns a NodePool that provisions Windows (amd64) nodes for the given +// nodeClass by replacing the default os=linux requirement with os=windows. +func (env *Environment) WindowsNodePool(nodeClass *v1beta1.AKSNodeClass) *karpv1.NodePool { + nodePool := env.DefaultNodePool(nodeClass) + coretest.ReplaceRequirements(nodePool, + karpv1.NodeSelectorRequirementWithMinValues{ + Key: v1.LabelOSStable, + Operator: v1.NodeSelectorOpIn, + Values: []string{string(v1.Windows)}, + }, + ) + return nodePool +} + // Pod wraps coretest.Pod for Azure E2E tests; use it instead of coretest.Pod when the test should apply Azure environment defaults. // Currently this is any time one has to work around taint race described in https://github.com/Azure/karpenter-provider-azure/issues/1625 // and cannot use Deployment instead. diff --git a/test/suites/windows/suite_test.go b/test/suites/windows/suite_test.go new file mode 100644 index 0000000000..d069276fe8 --- /dev/null +++ b/test/suites/windows/suite_test.go @@ -0,0 +1,128 @@ +/* +Portions Copyright (c) Microsoft Corporation. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package windows_test + +import ( + "fmt" + "testing" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + karpv1 "sigs.k8s.io/karpenter/pkg/apis/v1" + "sigs.k8s.io/karpenter/pkg/test" + + "github.com/Azure/karpenter-provider-azure/pkg/apis/v1beta1" + "github.com/Azure/karpenter-provider-azure/pkg/utils/zones" + "github.com/Azure/karpenter-provider-azure/test/pkg/environment/azure" +) + +var env *azure.Environment + +// windowsPauseImage is a multi-arch pause image with a Windows Server 2022 (ltsc2022, +// build 20348) variant, matching the Windows2022 node we provision. +const windowsPauseImage = "mcr.microsoft.com/oss/kubernetes/pause:3.9" + +func TestWindows(t *testing.T) { + RegisterFailHandler(Fail) + BeforeSuite(func() { + env = azure.NewEnvironment(t) + }) + AfterSuite(func() { + env.Stop() + }) + RunSpecs(t, "Windows") +} + +var _ = BeforeEach(func() { env.BeforeEach() }) +var _ = AfterEach(func() { env.Cleanup() }) +var _ = AfterEach(func() { env.AfterEach() }) + +var _ = Describe("Windows", func() { + BeforeEach(func() { + // Windows nodes are only provisionable via the AKS Machine API provision mode. + if !env.IsAKSMachineAPIMode() { + Skip("Windows node provisioning is only supported in AKS Machine API provision mode") + } + // Windows machine names are bounded by the Windows NetBIOS computer-name limit once the AKS RP + // composes the VM name from the pool and machine names. The reserved NAP pool ("aksmanagedap") + // uses VM name "aks" (machine <= 12); a custom/self-hosted machines pool uses + // "aks-" (machine <= 5) and additionally requires the pool name to be <= 6 chars. + // Karpenter sizes the Windows machine name to the pool (see WindowsMachineNameMaxLength), so this + // test runs against the reserved NAP pool OR a sufficiently short custom machines pool. + if env.MachineAgentPoolName != "aksmanagedap" && len(env.MachineAgentPoolName) > 6 { + Skip(fmt.Sprintf("Windows machines require the reserved aksmanagedap pool or a custom machines pool name <= 6 chars; got %q (%d chars)", + env.MachineAgentPoolName, len(env.MachineAgentPoolName))) + } + }) + + It("should provision a Windows node and run a Windows pod", func() { + nodeClass := env.WindowsNodeClass() + nodePool := env.WindowsNodePool(nodeClass) + // Keep the test as simple and portable as possible: pin to the regional (zoneless) + // offering so it provisions in subscriptions/regions without availability-zone support. + // The SKU itself is intentionally left unconstrained: in AKS Machine API mode Karpenter + // requests a Gen2 Windows image (UseWindowsGen2VM) whenever the selected SKU supports it, + // so Windows provisions on any Hyper-V generation, including Gen2-only sizes. + test.ReplaceRequirements(nodePool, karpv1.NodeSelectorRequirementWithMinValues{ + Key: corev1.LabelTopologyZone, + Operator: corev1.NodeSelectorOpIn, + Values: []string{zones.Regional}, + }) + + deployment := test.Deployment(test.DeploymentOptions{ + Replicas: 1, + PodOptions: test.PodOptions{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{"app": "windows-inflate"}, + }, + Image: windowsPauseImage, + NodeSelector: map[string]string{ + corev1.LabelOSStable: string(corev1.Windows), + }, + // Tolerate the OS taint Karpenter may surface during Windows node registration. + Tolerations: []corev1.Toleration{{ + Key: corev1.LabelOSStable, + Operator: corev1.TolerationOpEqual, + Value: string(corev1.Windows), + Effect: corev1.TaintEffectNoSchedule, + }}, + ResourceRequirements: corev1.ResourceRequirements{ + Requests: corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("500m"), + corev1.ResourceMemory: resource.MustParse("256Mi"), + }, + }, + }, + }) + + env.ExpectCreated(nodeClass, nodePool, deployment) + + // Windows nodes take noticeably longer to provision and pull images than Linux. + pods := env.EventuallyExpectHealthyDeploymentWithTimeout(25*time.Minute, deployment) + env.ExpectCreatedNodeCount("==", 1) + + node := env.GetNode(pods[0].Spec.NodeName) + Expect(node.Labels).To(HaveKeyWithValue(corev1.LabelOSStable, string(corev1.Windows))) + Expect(node.Labels).To(HaveKeyWithValue(v1beta1.AKSLabelOSSKU, v1beta1.OSSKUWindows2022)) + Expect(node.Labels).To(HaveKeyWithValue(karpv1.NodePoolLabelKey, nodePool.Name)) + }) +})