Skip to content

Upgrade the Temporal Worker Controller to 1.12 - #1194

Merged
duynhne merged 1 commit into
mainfrom
chore/temporal-worker-controller-1.12
Oct 2, 2026
Merged

duynhne merged 1 commit into
mainfrom
chore/temporal-worker-controller-1.12

Conversation

@duynhne

@duynhne duynhne commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

What

This upgrades both Worker Controller charts from 0.28.0 to 0.31.0 (controller 1.9.0 → 1.12.0). The owner picked this from the SDK and server follow-ups in #1107.

From the official notes (v1.10.0, v1.11.0, v1.12.0):

  • 1.12:
    • prunes superseded inactive versions;
    • surfaces poller and gate-workflow health as conditions and events;
    • backs off failing drained-version deletes;
    • no longer treats an already-current version as a rollback;
    • adds a CA-injector Job option.
  • 1.11:
    • scales draining versions back up from 0;
    • removes the cert-manager subchart.
  • 1.10:
    • deletes rendered autoscalers at sunset;
    • prunes a version before deleting its Deployment;
    • is more resilient to inconsistent Temporal state;
    • adds the ClusterConnection CRD.

cert-manager: this platform always had certmanager.install: false, so the upgrade guide's migration does not apply. I dropped the now-dead install key. Our values render identically with and without it on 0.31.0. The webhook certificate (temporal-worker-controller-serving-cert → Secret webhook-server-cert, CA injected into the WorkerResourceTemplate webhook) is still issued by the platform's cert-manager.

Render diff, 0.28.0 vs 0.31.0, with our values:

  • RBAC for clusterconnections and clusterconnections/finalizers;
  • env WRT_HPA_MATCH_LABELS_STRIP_TEMPORAL_PREFIX=false;
  • no hooks.

Verified on Kind (branch overlay)

Check Result
HelmReleases crds upgraded to 0.31.0 first (dependsOn), then the manager; image temporalio/temporal-worker-controller:v1.12.0
WorkerDeployments order/order-fulfillment (current 2.10.1-fcc8) and checkout/checkout-abandon (current 0.13.1-4955): ConnectionHealthy=True, Ready=True, RolloutComplete=True
WorkerResourceTemplates and KEDA both scaler WRTs Ready=True; both ScaledObjects Ready
Manager log no errors in the 5 min after the upgrade
make e2e-saga GATE=kind SG.1–SG.4 PASS (SG.4: Pinned on the Current build, no ramp)

Draft until the rollout drill runs. The G4 release (order and checkout on SDK 1.49) pins new worker builds on Kind. That gives a real Progressive rollout (10 → 50 → 100) on this controller, and lets me see the superseded version drain and get pruned. I'll add that evidence here before marking this ready.

Both charts 0.28.0 -> 0.31.0, controller 1.9.0 -> 1.12.0. 1.12 prunes
superseded inactive versions, surfaces poller and gate-workflow health
as conditions, and stops treating an already-current version as a
rollback; 1.11 scales draining versions back up from zero.

Chart 0.30.0 removed the cert-manager subchart. This platform never
installed it (certmanager.install: false), so no migration is needed;
the now-dead install key is dropped and the rendered manifests are
identical. The webhook certificate is still issued by the platform's
cert-manager through the chart's self-signed Issuer.
@duynhne

duynhne commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Rollout drill on controller 1.12 (from #1196's pins)

The real Progressive rollout ran on Kind with this branch's controller overlaid. The controller log:

Time (+07) Worker Step
14:56:42 order-fulfillment applying ramp 10% → 2.10.2-8d45
14:57:12 order-fulfillment applying ramp 50%
14:57:43 order-fulfillment registering new current version 2.10.2-8d45; 2.10.1-fcc8 → Draining
14:56:49 / 14:57:19 / 14:57:50 checkout-abandon the same three steps → current 0.13.2-f678; 0.13.1-4955 → Draining
  • Status: ConnectionHealthy=True, Ready=True, RolloutComplete=True on both.
  • Pollers: an ActivePollers event for both versions during the ramp.
  • Gate: make e2e GATE=kind passes all rows, SG.4 (Pinned on the new Current build) included.
  • KEDA: the new version's ScaledObject logged Worker Deployment Version not found once, at 07:56:42Z, before the version registered. There were no errors after that, and KedaScaledObjectErrors cleared by itself.

Still to observe: pruning of the superseded versions. The spec has sunset.scaledownDelay: 1h and deleteDelay: 0s, so the drained versions scale down about an hour after they drain, and then their Deployments are deleted. I'll add that read-back here and then mark this ready.

duynhne added a commit that referenced this pull request Oct 2, 2026
Both releases move to Temporal Go SDK 1.49.0 and passed the full
local-stack release audit (A/B/C + C22) on their merged SHAs. The
worker pins start a Progressive rollout through the Worker Controller,
which is the rollout drill for #1194.
@duynhne

duynhne commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Version pruning, observed on controller 1.12

About an hour after promotion (sunset.scaledownDelay: 1h, deleteDelay: 0s), the controller log shows:

Time (+07) Worker Step
16:05:29–16:06:00 order-fulfillment deleted worker resource on version sunset (the old version's ScaledObject)
16:06:00 order-fulfillment deleted deprecated worker deployment version 2.10.1-fcc8, deleting deployment
16:06:07 checkout-abandon deleted deprecated worker deployment version 0.13.1-4955, deleting deployment

Afterwards:

  • Each WorkerDeployment has deprecatedVersions: [].
  • Only order-fulfillment-2-10-2-8d45 and checkout-abandon-0-13-2-f678 remain, each 1/1.
  • One ScaledObject per worker is left, both Ready.

That completes the drill: ramp 10 → 50 → current, drain, scale-down and prune all ran on 1.12. Ready for review.

@duynhne
duynhne marked this pull request as ready for review October 2, 2026 09:06
@duynhne
duynhne requested a review from duyhenryer as a code owner October 2, 2026 09:06
@duynhne
duynhne merged commit 6da4145 into main Oct 2, 2026
7 checks passed
@duynhne
duynhne deleted the chore/temporal-worker-controller-1.12 branch October 2, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant