design: Node Image Version Pinning in Karpenter by rakechill · Pull Request #1720 · Azure/karpenter-provider-azure

rakechill · 2026-06-10T19:11:07Z

Summary

Design document for adding imageVersion pinning support to AKSNodeClass, allowing customers to pin node image selection to a specific AKS node image version (e.g., 202604.24.0).

Motivation

Karpenter users can set imageFamily but cannot currently:

Pin to a known-good version when the latest image has a regression
Decouple node image updates from Kubernetes version upgrades
Stay on a stable image version for a bounded period

The only workaround today is disabling Karpenter entirely and falling back to AKS autoscaling groups.

Key Design Points

New imageVersion field on AKSNodeClassSpec:
- Pins node image selection to a specific version
- Validated at admission via regex ^\d{6}\.\d{2}\.\d+$ (current YYYYMM.DD.patch format only)
imageCreateDate per image in status.images[]:
- Immutable release date stored once at reconciliation time
- Consumers compute age on read, avoiding daily status churn
ImageWithinSupportWindow status condition:
- Warning-only (never blocks provisioning)
- Not a readiness-dependent condition
karpenter_image_age_days gauge metric:
- Computed at scrape time from imageCreateDate
Cache key fix:
- imageVersion is included in cacheKey()
- Pin changes force a fresh lookup so status reflects .spec.imageVersion correctly
Maintenance window behavior with pinning:
- When imageVersion is set, pin is authoritative
- Reconciler skips maintenance-window update gating/override logic for latest selection
Drift detection:
- Continues to work via existing status.images[] vs node image comparison
AKSNodeClassSpec.ImageID cleanup:
- Existing hidden/unused stub is removed as part of this design
Alternatives considered:
- do-not-disrupt annotation
- EKS-style amiSelectorTerms pattern
- NodePool.expireAfter
- All evaluated against rollback, k8s/image decoupling, and bounded stability-window goals

Open Questions (pending Node SIG response)

Is there a supported compatibility contract between Kubernetes version and node image version?
If a user pins a node image version and upgrades Kubernetes, should pinning across that upgrade be allowed, blocked, or explicitly validated?
On Kubernetes upgrades that trigger image refresh (ImagesReady=False), should pinned versions be re-validated against the new Kubernetes version?
Is ListNodeImageVersions presence the intended compatibility signal, or is there another authoritative source?

- Reframe support window without hardcoding 90-day duration - 90 days is now a warning threshold (matching AppLens), not policy - Add 'Path to enforcement' explaining how to promote condition to readiness dependent - Add open questions for threshold configurability and enforcement strategy - Fix SLA references to use 'AKS support window' throughout

- Move Prepared Image Spec background to section 6 (future extensibility) - Replace imageAge (metav1.Duration) with imageAgeDays (int32) for readability - Clarify image version existence validation flow across definitions - Fix relationship map alignment - Renumber all sections and cross-references

AKS image versions exist in two observed formats: - Current: YYYYMM.DD.patch (e.g. 202512.06.0) - Legacy: YYYY.MM.DD (e.g. 2022.10.03) The existing isNewerVersion() in nodeimageversionsclient.go already handles both (added in commit 3a9cbc0, PR #526). The admission regex and age-parsing description are updated to match.

rakechill · 2026-06-15T22:40:40Z

Finding: `imageVersion` regex should accept both AKS version formats

While cross-referencing the design against the existing codebase, we found that AKS image versions exist in two distinct formats, not just one:

Format	Example	Used for
`YYYYMM.DD.patch`	`202512.06.0`	Current AKS releases (since ~2023)
`YYYY.MM.DD`	`2022.10.03`	Legacy AKS releases

The original design had a single-format regex (^\d{6}\.\d{2}\.\d+$) that would silently reject legacy version strings as valid pin targets — for example, a customer attempting to rollback to an older image using a YYYY.MM.DD version would get an admission error.

Evidence in the codebase:

isNewerVersion() in pkg/providers/imagefamily/nodeimageversionsclient.go was introduced in PR feat: ListNodeImageVersions + shared image gallery support #526 (commit 3a9cbc0d) specifically to handle both formats, with a comment: "the new versioning scheme is yearmm.dd.build, previously it was yy.mm.dd without the build id".
Tests in pkg/providers/imagefamily/nodeimageversionsclient_test.go cover cross-format comparisons like 202411.12.0 vs 2022.12.15.
The fake API in pkg/fake/nodeimageversionsapi.go still emits 2022.10.03-style versions.

Fix applied: Updated the admission regex to an OR pattern:

!!:s^(\d{6}\.\d{2}\.\d+|\d{4}\.\d{2}\.\d{2})$

This matches both formats at admission time. Gallery existence validation in the status reconciler remains the authoritative check regardless of format.

- Cache key: require imageVersion in cacheKey() to force immediate refresh on pin set/change (was silently stale for up to 3 days) - Non-goals: AKS Machine API + CIG pinning is explicitly out of scope; CIG + bootstrapping client path works naturally via full ID comparison - Drift table: document per-path drift support clearly, call out AKS Machine API + CIG as non-goal - Age parsing: add two-branch pseudocode for YYYYMM.DD.patch vs YYYY.MM.DD formats, following isNewerVersion() pattern - Support window threshold: resolve as controller flag (--image-support-window-warning-days, default 90), not Helm value, so self-hosted users can configure it too

matthchr · 2026-06-15T20:51:11Z

+|---|---|
+| **In-window** | Image age ≤ 90 days from release date. No restrictions; full support. |
+| **Outside-supported-window** | Image age > 90 days. Support may require upgrading to a newer image as a prerequisite for resolution. |
+| **Unsupported** | Potential future enforcement state where specific operations (e.g., scale-up) could be blocked. Not part of initial rollout. |


This is something we need to talk/work with node sig on, but I saw Jorge pushing back on this and I agree with that.

I guess not critical here as we're punting on it though.

Yes, I already discussed with node sig folks and PMs (including Jorge). They are still ironing out the exact long-term policy, but are aligned with adding the 90 days warning at the moment.

matthchr · 2026-06-15T20:51:40Z

+1. AKS ships node images frequently: **weekly for Linux, monthly for Windows**.
+2. Customers are expected to update regularly — via auto-upgrade channels or manual upgrades.
+3. **Rollback and pinning are supported flexibility mechanisms**, but they **do not override lifecycle expectations**. They are meant for short-term operational recovery, not long-term avoidance of updates.
+4. Within the supported window, scale-up is expected to be **unconditionally allowed** regardless of cluster state.


and at least for now, outside of the supported window scale up is also allowed?

Can you clarify that here in the doc?

matthchr · 2026-06-15T20:57:24Z

+If we expose a version-pinning mechanism, it should:
+
+- Allow customers to pin within the supported window.
+- Surface image age as observable state (e.g., status conditions).


Age, or publish date?

Is there a pattern for computing an always-increasing counter in k8s resources? I think usually you see something like: lastUpdatedAt, rather than timeSinceLastUpdated.

Ahh I see you propose imageAgeDays, which we could do, but I'd suggest doing a date (2026-06-12) or datetime (2026-06-12 00:00:00).

I think that's more standard?

AI agrees:
🤖

Image Creation Date vs. Image Age in Days

Option 1: imageCreateDate: 2026-06-12 is the better choice for a CRD field.

Why Creation Date wins:

Aspect Creation Date Age in Days

Staleness Always accurate — immutable fact Stale immediately after being written; requires continuous reconciliation to stay correct

Reconciliation cost No updates needed Must be updated daily or becomes wrong

Consistency Same value regardless of when you read it Value depends on when it was last reconciled vs. when you read it

Conflict/churn No status churn Daily updates cause unnecessary resource version bumps, watch events, and potential conflicts

Expressiveness Consumer can compute age, compare dates, bucket by month, etc. Loses the original date; can't derive it back without knowing when the field was last updated

Time zones / clock skew Standard metav1.Time or date string — well-understood Integer that's only meaningful relative to "now", which differs across controllers

Kubernetes convention Matches patterns like creationTimestamp, lastTransitionTime No precedent in upstream APIs for "age in days" fields

When age might be preferable:

If you want a user-facing display hint (like kubectl get columns showing "3d") — but that's better done in a printer column computed from the date.

If the source data genuinely only provides age without an absolute timestamp (rare).

Recommendation:

Store the creation date (metav1.Time or a date string). Let consumers (controllers, CEL validation, printer columns) compute age on read. This follows the same principle as ConditionTypeConsistentStateFound in the file you're looking at — conditions store lastTransitionTime, not "duration since last transition."

Yeah that makes sense

TODO: update this to be a static date rather than calculated days since

matthchr · 2026-06-15T21:00:56Z

+        │
+        ▼
+AKSNodeClass.status.images[]         (reconciled by nodeclass status controller)
+        │                             e.g. [{ ID: "/CommunityGalleries/.../versions/202604.24.0",


Note that this is resolved from the public API https://learn.microsoft.com/en-us/rest/api/aks/container-service/list-node-image-versions?view=rest-aks-2026-04-02-preview&viewFallbackFrom=rest-aks-2026-03-02-preview&tabs=HTTP

ack, will reflect

matthchr · 2026-06-16T00:28:27Z

+When `imageVersion` is **unset** (default), drift detection works as
+today — comparing against the latest available version.
+
+No changes are needed to `isImageVersionDrifted()` itself — it already


We will need to make sure that the Status controller takes the .spec field for pinning into account when doing the 72h cache thing it does.

TODO: add this detail into the design

matthchr · 2026-06-16T00:30:19Z

+- Prepared Image Spec operates **outside** the AKS-managed galleries —
+  it references a customer-owned image resource.
+
+A well-shaped API keeps these as distinct, non-overlapping fields


matthchr · 2026-06-16T00:33:41Z

+`ImagesReady` to `False`, which triggers a full image refresh
+(Scenario A, case 2 in the existing code).
+
+**When `imageVersion` is set**, this refresh should still resolve to the


Does this mean the nodes roll or they don't roll?

I think they have to roll? Technically they can roll to the same image but they have to roll.

They'll still roll, but node image (likely) shouldn't change -- still need to check w/ node sig on this

TODO: add that the nodes still get upgraded, but same image version

matthchr · 2026-06-16T00:34:28Z

+included to demonstrate that the API shape in section 5 is
+forward-compatible with upcoming AKS capabilities.
+
+### 6.0 Background: Prepared Image Specification


Good we have this -- just noting I didn't review it as closely.

Yeah, I'm still deciding if adding this as a secondary (related) design might be better. Maybe a different PR left in draft until we return to this. I think @comtalyst's meeting this week will help inform/shape this design, too.

matthchr · 2026-06-16T00:34:56Z

+Remaining questions not covered by the design sections above:
+
+### Operational
+1. Should we emit Kubernetes events when a pinned version crosses the


Where/on what resource?

Probably AKSNodeclass.

matthchr · 2026-06-16T16:18:31Z

+
+## 1. Background: The Node Image Support Window
+
+AKS is formalizing its **support window for node images**. While no


after review: We also don't discuss different options here. I personally tend towards pinning as you have proposed here, because I think it solves the problem as well as some other user complaints as well, but we should at least have some discussion/investigation about what other possible solutions look like, where we enable the user to control when the node image version is applied, but:

Don't let them pick the specific version

Don't let them pin across k8s version updates

Maybe in the appendix/at the end we can discuss those options at a high level and why we feel that they do not suit?

rakechill added 5 commits May 21, 2026 21:39

Add design doc for node image version pinning

bd6ad0b

design: remove resolved compatibility open questions

9dab2b2

Merge branch 'main' into rakechill/node-image-pinning-design

b672251

rakechill marked this pull request as ready for review June 10, 2026 19:12

rakechill added 2 commits June 11, 2026 13:35

Merge branch 'main' into rakechill/node-image-pinning-design

ffc89fe

matthchr reviewed Jun 16, 2026

View reviewed changes

rakechill added 5 commits June 19, 2026 23:25

address smaller PR feedback

54c43ef

only need newer image id syntax

c312331

remove prepared image spec design details

faecdac

add alternatives section + update open questions section

fc92aab

more details for amiselector

f6832ff

rakechill requested a review from matthchr June 20, 2026 00:34

Aspect	Creation Date	Age in Days
Staleness	Always accurate — immutable fact	Stale immediately after being written; requires continuous reconciliation to stay correct
Reconciliation cost	No updates needed	Must be updated daily or becomes wrong
Consistency	Same value regardless of when you read it	Value depends on when it was last reconciled vs. when you read it
Conflict/churn	No status churn	Daily updates cause unnecessary resource version bumps, watch events, and potential conflicts
Expressiveness	Consumer can compute age, compare dates, bucket by month, etc.	Loses the original date; can't derive it back without knowing when the field was last updated
Time zones / clock skew	Standard `metav1.Time` or date string — well-understood	Integer that's only meaningful relative to "now", which differs across controllers
Kubernetes convention	Matches patterns like `creationTimestamp`, `lastTransitionTime`	No precedent in upstream APIs for "age in days" fields


		## 1. Background: The Node Image Support Window

		AKS is formalizing its support window for node images. While no

Uh oh!

Conversation

rakechill commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Key Design Points

Open Questions (pending Node SIG response)

Related

Uh oh!

rakechill commented Jun 15, 2026

Finding: imageVersion regex should accept both AKS version formats

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Image Creation Date vs. Image Age in Days

Why Creation Date wins:

When age might be preferable:

Recommendation:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rakechill commented Jun 10, 2026 •

edited

Loading

Finding: `imageVersion` regex should accept both AKS version formats