Topology spread constraints fail with empty zone domain in counts

[justinwalz]

Version

Karpenter version: 1.6.6, and 1.7.1
Mode: Self hosted (see below comment, someone reproduced with NAP as well).
Kubernetes version: 1.33
Region: eastus2

Expected Behavior

With topology spread constraint maxSkew: 1 and whenUnsatisfiable: DoNotSchedule, Karpenter should provision nodes in zones that satisfy the constraint.

Current distribution: zone1=2, zone2=1, zone3=1
Expected: Provision in zone2 or zone3 (maxSkew would be 1)

Actual Behavior

Karpenter reports topology constraint as unsatisfiable and refuses to provision nodes. Logs show an unexpected empty string domain in topology counts:

counts = : 0, eastus2-1: 2, eastus2-2: 1, eastus2-3: 1

This appears to cause incorrect maxSkew calculation. My guess here is that the first part of that error string is missing a zone, hence : 0, when others have format $zone: $count.

Steps to Reproduce the Problem

1. Deploy with topology spread constraints in a zonal region:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: my-name

2. Scale to create pending pods requiring new nodes

3. Observe Karpenter refusing to provision despite valid zones available

Resource Specs and Logs

Note, we have two node pools, default and ephemeral (spot instances for cronjob job pods).

In both cases for us, one Deployment, and one Statefulset, both using the default nodepool, caused this error.

{"level":"ERROR","time":"2026-01-27T18:23:45.982Z","logger":"controller","message":"could not schedule pod","commit":"28204ca","controller":"provisioner","namespace":"","name":"","reconcileID":"7b152da1-16b0-41a1-a483-025bfe3f72ca","Pod":{"name":"<REDACTED_POD_NAME>","namespace":"<REDACTED_POD_NAMESPACE>"},"taint":"dedicated=ephemeral:NoSchedule","error":"did not tolerate taint (taint=dedicated=ephemeral:NoSchedule); unsatisfiable topology constraint for topology spread, key=topology.kubernetes.io/zone (counts = : 0, eastus2-1: 1, eastus2-2: 1, eastus2-3: 1, podDomains = topology.kubernetes.io/zone Exists, nodeDomains = topology.kubernetes.io/zone Exists","errorCauses":[{"error":"did not tolerate taint (taint=dedicated=ephemeral:NoSchedule)"},{"error":"unsatisfiable topology constraint for topology spread, key=topology.kubernetes.io/zone (counts = : 0, eastus2-1: 1, eastus2-2: 1, eastus2-3: 1, podDomains = topology.kubernetes.io/zone Exists, nodeDomains = topology.kubernetes.io/zone Exists"}]}

Short-term mitigation

Switching from DoNotSchedule to ScheduleAnyway allowed Karpenter to create a node, and immediately following, saw the event Pod should schedule on: nodeclaim/$name.

I include this because it seems to me that this setting is only really for the kube scheduler, and since karpenter still needed to create a new node to fit, it probably should have been able to do so in the earlier case with DoNotSchedule. In fact, we rely on DoNotSchedule to require karpenter to schedule a new node in another zone, so that we maintain zone spread.

This is what leads me to believe this is a bug in the implementation somewhere.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[justinwalz]

I'd like input from a maintainer here, but so far thinking maybe:

This file

• pkg/providers/instancetype/instancetypes.go

Karpenter core

• pkg/controllers/provisioning/scheduling/ and pkg/scheduling/
• Locations with topology.kubernetes.io/zone: "" in offering requirements seem to register an empty string as a valid topology domain, creating the ": 0" entry in counts

[k24dizzle]

I was able to reproduce this in my cluster with Node AutoProvisioning enabled. I see very similar behavior and as a result only see 3/5 pods able to schedule.

Failed to schedule pod, unsatisfiable topology constraint for topology spread, key=topology.kubernetes.io/zone (counts = : 0, westus2-1: 1, westus2-2: 1, westus2-3: 1, podDomains = topology.kubernetes.io/zone Exists, nodeDomains = topology.kubernetes.io/zone In [westus2-1 westus2-2 westus2-3];

I think this stems from the fact that Azure offers instance types that do not have a topology.kubernetes.io/zone label set. And for some reason Karpenter thinks that an empty zone "" is a valid domain which it calculates TSC's against. If there is a way to ignore these non-zonal instances when listing the instance types for Azure, I think it would help.

YAMLs

---
apiVersion: karpenter.azure.com/v1beta1
kind: AKSNodeClass
metadata:
  name: tsc-repro-node
  annotations:
    kubernetes.io/description: AKSNodeClass for TSC deadlock reproduction
spec:
  imageFamily: Ubuntu2204
  osDiskSizeGB: 40

---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: tsc-repro
  annotations:
    kubernetes.io/description: NodePool for TSC deadlock reproduction
spec:
  template:
    metadata:
      labels:
        k8s.tmprl.cloud/node-type: tsc-repro
    spec:
      nodeClassRef:
        group: karpenter.azure.com
        kind: AKSNodeClass
        name: tsc-repro-node
      taints:
      - key: k8s.tmprl.cloud/node-type
        value: tsc-repro
        effect: NoSchedule
      startupTaints:
      - key: node.cilium.io/agent-not-ready
        value: "true"
        effect: NoExecute
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: [on-demand]
      - key: kubernetes.io/arch
        operator: In
        values: [amd64]
      - key: karpenter.azure.com/sku-cpu
        operator: Lt
        values: ["9"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30m
    budgets:
    - nodes: "1"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tsc-repro
spec:
  replicas: 5
  selector:
    matchLabels:
      app: tsc-repro
  template:
    metadata:
      labels:
        app: tsc-repro
    spec:
      nodeSelector:
        k8s.tmprl.cloud/node-type: tsc-repro
      tolerations:
      - key: k8s.tmprl.cloud/node-type
        value: tsc-repro
        effect: NoSchedule
      - key: node.cilium.io/agent-not-ready
        value: "true"
        effect: NoExecute
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: tsc-repro
            topologyKey: kubernetes.io/hostname
      topologySpreadConstraints:
      - maxSkew: 1
        minDomains: 3
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: tsc-repro
      containers:
      - name: app
        image: busybox:1.36.1
        command: ["sh", "-c", "sleep infinity"]
        resources:
          requests:
            cpu: "2500m"
            memory: "1Gi"

I attempted a fix here but still need to test it out.

[tallaxes]

Need to take a closer look, but Yes I suspect the presence of non-zonal nodes confuses the scheduler here. See if these notes (especially second and third point) help:

• Note that topology spread constraint scheduling logic does not have intrinsic knowledge of possible domain values, e.g. it does not know how many / which zones are available, beyond observing the corresponding values on existing Nodes.
• Note that non-zonal (zone=0) agent pools in cluster (if any) essentially add another domain value that topology spread constraint on zone will be taking into account. This may explain certain apparent scheduling or scaling failures. Using zonal agent pools instead avoids this confusion.
• Alternatively, to prevent scheduler (and Karpenter) from even considering the non-zonal (zone=0) agent pool in minDomains / maxSkew constraints, one can taint those agent pools and specify nodeTaintPolicy=Honor, or specify additional node affinity in the workload with an explicit list of expected zones (nodeAffinityPolicy=Honor by default).
• To force the use of all zones one should include minDomains=3 (equal to the total number of zones). In combination with maxSkew, this will force NAP/Karpenter to provision additional nodes in the right zones. Without minDomains, scheduler & Karpenter can continuously satisfy minSkew with less then 3 zones.
• We should strongly encourage the use of whenUnsatisfiable: DoNotSchedule. Depending on goals, one could achieve reasonable results with ScheduleAnyway, but these constraints can ultimately be ignored by scheduler (which will find ways to schedule workloads onto existing nodes without causing scaleup) and NAP/Karpenter (which can make - or skip - scale up decisions that would appear surprising). Also note that gradual relaxation of soft preference constraints are expensive in scheduling simulations, and going forward we may support a more performant mode that always ignores them.
• Note that none of these guarantees (or can guarantee) maintaining topology spread over time, e.g. in the presence of failures.

[justinwalz]

Hi @tallaxes, thank you for the response here.

For bullet point 2, we do not use non-zonal agent pools. Our setup:

• System Pool (AKS-managed), 3 nodes in VMSS, one per zone (eastus2-1, -2, -3)
  • Zonal, and all 3 nodes have proper zone labels
  • Tainted with CriticalAddonsOnly=true:NoSchedule, workloads will not run here unless they tolerate it, and our karpenter pods do, so they run here. Remainder of all nodes are managed by karpenter directly.
• Default Pool (Karpenter-managed)
  • Currently has nodes across all 3 zones, each with valid labels for topology.kubernetes.io/zone, all on demand instances, in SKU families D, E, F

Please confirm that satisfies the case here.

For bullet point 3, I believe this is also a non-issue for us.

• system pool is already zonal, no empty domain
• system pool is already tainted CriticalAddonsOnly, and nodeTaintPolicy defaults to Honor for DoNotSchedule constraints, so those nodes are already excluded from topology math

Please also confirm I'm correctly understanding your point.

For bullet 4, minDomains: 3, we don't have this in our topospread settings but we do already have node across all three zones, so we can add it, but I don't think it'll change anything yet.

[k24dizzle]

I did a bit more thinking, and wanted to jot down my thoughts.

Background

If Karpenter sees an instance with topology.kubernetes.io/zone = "", Karpenter's provisioning logic treats "" as a valid topology domain, confusing topology spread constraints (TSCs). My PR tries to address this.

This also seems broader than just Karpenter but also Kubernetes. If applications are deployed across regions and use zonal TSCs, empty-zone semantics here seem like they could break those workloads. How can we spread pods across zones if there is an empty zone?

Idea to fix

Maybe we could move the fix to the karpenter-provider-azure repository instead, and not return any instances with a topology.kubernetes.io/zone = "".

There seems to be two cases where karpenter-provider-azure returns instance types to Karpenter with an empty zone value based on this logic here.

1. The instance type is genuinely non-zonal in that region
2. The instance type is zonal in that region, but all zones are restricted/unavailable for the subscription

However, I don't think we want Karpenter to even consider these kinds of instance types/SKUs.

Non-zonal SKU example

az vm list-skus --location northcentralus --resource-type virtualMachines --size Standard_A1_v2 --output json | jq '.[0] | {locationInfo, restrictions}'

{
  "locationInfo": [
    {
      "location": "northcentralus",
      "zoneDetails": [],
      "zones": []
    }
  ],
  "restrictions": []
}

In this case, the SKU seems genuinely non-zonal for the region, which seems valid, but it still should not result in a phantom "" zone domain.

In practice, I see these non-zonal nodes get assigned topology.kubernetes.io/zone = "0". When we generate the instance types zones here instead of an empty string, could we set this to sets.New("0")?

% kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}{end}'
...
aks-system-24644131-vmss000000	0
aks-system-24644131-vmss000001	0
aks-system-24644131-vmss000002	0

• In my case though, I don't want Karpenter to even consider provisioning a non-zonal instance type if my workloads use TSCs that want spread it across zones. I don't want to spread 4 pods across 3 AZs and 1 non-zonal domain.
• It would be nice if there was a way where we could tell the karpenter-provider-azure to not offer these instance types to the Karpenter controller.

Restricted zonal SKU example

az vm list-skus --location westus2 --resource-type virtualMachines --size Standard_D15_v2 --output json | jq '.[0] | {locationInfo, restrictions}'

{
  "locationInfo": [
    {
      "location": "westus2",
      "zoneDetails": [],
      "zones": ["2", "3", "1"]
    }
  ],
  "restrictions": [
    {
      "reasonCode": "NotAvailableForSubscription",
      "restrictionInfo": {
        "locations": ["westus2"],
        "zones": ["1", "2", "3"]
      },
      "type": "Zone",
      "values": ["westus2"]
    }
  ]
}

In this case, the SKU appears zonal for the region, but all zones are restricted for the subscription. My expectation is that these offerings should be filtered out or marked unavailable rather than surfaced as an empty-zone offering.

[k24dizzle]

I went ahead and gave this a shot here: #1613 (to omit the restricted SKUs)

[tallaxes]

Took another look at the issue, and Yes empty "" is breaking TSC. Appreciate reporting, and experimenting with solutions - and apologies for the delay in addressing this. Fixing this by altogether filtering regional-only VMs (in either core or provider) is likely not the right approach, as we would want to support regional-only SKUs (there are valid use cases) - and regional provisioning in general.

In practice, I see these non-zonal nodes get assigned topology.kubernetes.io/zone = "0". When we generate the instance types zones here instead of an empty string, could we set this to sets.New("0")?

That is the right approach, aligns with Azure/AKS and fixes TSC. #1615 should fix it - and pave the way for better supporting provisioning of regional-only and regional VMs in general (which I am also taking a look at, there are valid use cases).