[Release tracker] TauGrid v0.4.2 deployment: portal refresh, bug fixes, and open-issue closure

[chokevin]

User problem

We need one deployment tracker for the next TauGrid release: capture the changes made since v0.4.0, deliver the refreshed TauGrid Portal experience alongside bug fixes, and account for every originally tracked issue before release closeout.

Desired outcome

Ship and deploy TauGrid v0.4.2 with an agreed release commit, published artifacts, a signed-off Portal refresh, and verified fixes. Each tracked bug needs a fix or an explicitly accepted deferral; an unchecked item is not implicitly dropped.

Version decision: This request began as v0.4.1. Target v0.4.2 because #243 prepared v0.4.1 and #253 subsequently bumped the unified release. v0.4.0 remains the latest published GitHub release at this triage.

Scope decision, 2026-09-09 PDT: All three tracked enhancements (#113, #92, #83) are deferred to the 0.4.3 milestone. They remain open and are not v0.4.2 launch blockers. Bug recommendations below are triage recommendations, not approved deferrals.

Accepted limitation, 2026-09-10 PDT: The release owner explicitly deferred automatic recovery from the upstream AzAPI asynchronous terminal-operation identity failure as a documented, non-blocking v0.4.2 limitation. Provisioning can fail visibly and require operator assessment; this is not a fix or a guarantee that every ADX initialization failure recovers automatically. Keep #161 open. This decision does not waive the Function waiter safety fix, actual permission-migration evidence, or combined-candidate acceptance. The original incident's HTTP-versus-LRO response shape remains unknown.

Proposed approach

1. Changes already merged since v0.4.0

Checked boxes in this section mean merged, not published, deployed, or release-accepted. Confirm inclusion in the selected release commit.

• Improve Portal navigation, telemetry accuracy, and Stellar workflows #252 and feat(portal): replace legacy UI with native React #264 - Portal UX foundations followed by the native React refresh with integrated experiment navigation, comparisons, and evidence.
• fix(run): validate workspace output paths during client dry-run #260, fix(cli): bind serve deploy to active workspace #262 and fix(serve): resolve workspace checkpoints and support explicit Deployment commands #263 - Workspace-bound dry-run/serving, checkpoint paths, and explicit Deployment command/argument support.
• fix(logs): discover historical RayJob ADX connections #254 - Historical RayJob ADX log connection discovery.
• feat(cli): add agent-readable run status #239 - Agent-readable CLI run status.
• Configure metrics offload in run files #235 - Metrics offload configuration in run files.
• Add workspace GPU chargeback and Flex GPU setup #234 - Workspace GPU chargeback and Flex GPU setup.
• feat(cli): add portable cluster credential adapters #225 - Portable cluster credential adapters.
• release: publish Windows tau CLI binaries #228 and test: validate CLI on Windows #229 - Windows CLI binaries and validation.
• Improve Tau CLI help and update vulnerable dependency #223 - CLI help improvements and dependency fix.
• docs(terraform): harden AKS deployment and verification #222 and test(terraform): preserve Bash waiter line endings #246 - AKS Terraform guidance and waiter line-ending coverage.
• fix(ray-image): upgrade to Ray 2.56.1 for protobuf 7 compatibility #233 - Ray 2.56.1 / protobuf 7 compatibility.
• charts: prepare TauGrid 0.4.1 dependency update #243 and release: bump TauGrid to 0.4.2 #253 - Unified release/chart preparation, including Kueue 0.19.2 and gpu-monitoring 0.1.8 with CRD retention and single-namespace contracts.
• chore(deps): bump the kubernetes-go-dependencies group across 6 directories with 3 updates #237 and chore(deps): update Azure Go SDK dependencies #242 - Kubernetes and Azure Go SDK dependency updates.
• chore(deps): bump github.com/NVIDIA/go-dcgm in /monitoring/gpu-health-checker #199, chore(deps): bump kube-state-metrics in /charts/adx-mon #214, chore(deps): bump ruff from 0.16.4 to 0.16.5 in /sdk/python/python in the python-minor-patch group across 1 directory #218, chore(deps): bump oss/go/microsoft/golang in /images/adx-mon #220, chore(deps): bump github/codeql-action/upload-sarif from 4.37.8 to 4.37.9 #230, chore(deps): bump oss/go/microsoft/golang in /images/adx-mon #245, chore(deps): bump ruff from 0.16.5 to 0.16.6 in /sdk/python/python in the python-minor-patch group #249, chore(deps): bump actions/deploy-pages from 5.0.0 to 5.0.1 #250, chore(deps): bump github.com/mattn/go-sqlite3 in /monitoring/gpu-health-checker #251, chore(deps): bump kube-state-metrics in /charts/adx-mon #255, chore(deps): bump golang.org/x/sys in /cli #256, chore(deps): bump github.com/NVIDIA/go-dcgm in /monitoring/gpu-health-checker #257 and chore(deps): bump modernc.org/sqlite in /portal #258 - Monitoring, tooling, runtime, and CI dependency maintenance.
• feat(site): add TauGrid blog #224, site: keep Hugo cache outside worktrees #238 and docs: add Discord community links #240 - Blog, documentation cache behavior, and community links.

2. Portal new look and bug-fix delivery

• Implement and merge the new look: feat(portal): replace legacy UI with native React #264, merged 2026-09-09 at 20:31 PDT.
• Obtain visual and functional acceptance on the exact release candidate; merging the implementation is not deployment signoff.
• Integrate Fix portal Job detail metadata matching #241 onto main, preserving current diagnostics and React section-recovery behavior. Merged as cdcd1df8; deployed-candidate acceptance remains below.
• Integrate fix(portal): keep metrics watch alive without scalars #232 onto main. Merged as 2ed3c292; deployed-candidate acceptance remains below.
• Exercise overview, workspace/run navigation, Job/RayJob details, logs, metrics, Stellar comparison/evidence, and GPU chargeback with representative deployed data.
• Confirm missing telemetry is unavailable rather than zero/healthy; check CPU intervals, GPU/memory availability, and cost coverage.
• Confirm Ray target/workspace isolation and supported read-only routes.
• Attach before/after visuals and a data-backed walkthrough; record limitations and acceptance.

3. Original issue inventory and dispositions

Original snapshot: 13 issues. Five are now closed, three enhancements explicitly deferred, and five bugs still open. The two open Portal fix PRs are tracked separately because they have no corresponding issue in this snapshot. A checked closed-issue box records GitHub closure, not live release acceptance.

Run, serve, and workspace fixes

• fix(run): --dry-run=client should validate output path against workspace root #212 - Client dry-run output paths: closed, fix(run): validate workspace output paths during client dry-run #260 merged.
• fix(serve): tau serve deploy requires namespace team label not set by TauWorkspace controller #210 - Serve namespace team-label mismatch: closed; fix(cli): bind serve deploy to active workspace #262 supplies the serve-specific workspace routing fix after fix(run): validate workspace output paths during client dry-run #260.
• fix(serve): --args flag doesn't support shell commands #209 - Explicit shell command/argument support: closed, fix(serve): resolve workspace checkpoints and support explicit Deployment commands #263 merged. Legacy --args behavior remains; Deployment uses explicit --command / --arg.
• fix(serve): --checkpoint path prefix doesn't match workspace output root #208 - Workspace checkpoint prefix: closed, fix(serve): resolve workspace checkpoints and support explicit Deployment commands #263 merged.
• [Bug]: WorkspaceDefaults.scratchMount is not in the TauWorkspace CRD and is silently pruned #111 - Unused client-side scratchMount field absent from the TauWorkspace CRD. Deferral recommendation below.
• Make project submission idempotent by client request ID #76 - Durable client-request-ID submission idempotency. Conditional deferral recommendation below.

ADX and infrastructure fixes

• adx-mon ManagementCommand does not report a completion condition #190 - ManagementCommand completion condition; mitigated installation sequencing is not schema-completion proof.
• adx-mon: retry Function reconciliation after transient ADX throttling #162 - Function reconciliation after transient ADX throttling; Terraform retry mitigation is not an upstream controller fix.
• Terraform: retry lifecycle recorder ADX principal assignment after managed identity propagation #161 - Lifecycle recorder ADX identity propagation. fix(terraform): retry lifecycle recorder ADX assignment #247 is merged; actual migration evidence remains open. Only automatic terminal-operation recovery is an accepted non-blocking v0.4.2 limitation.

Enhancements deferred to 0.4.3

• Record the producing tau version on generated workloads, and stamp payload-digest consistently #113 - Producing tau version and payload digest: assigned to milestone 0.4.3; left open.
• [Feature]: Support multiple active workspaces on one cluster #92 - Multiple active workspaces: assigned to milestone 0.4.3; left open.
• Add server-backed saved collections for cross-experiment curation #83 - Server-backed saved collections: assigned to milestone 0.4.3; left open.

Repository administration

• Action required: migrate or opt-out of migration to GitHub inside Microsoft #170 - Migration/opt-out action: GitHub issue closed; separate from product fixes.

4. Pre-launch bug triage

Initial source baseline: d8a4b022 (main including #264), 2026-09-09 PDT. Subsequent exact-candidate qualification at cdcd1df8 and scope decisions are recorded in this issue's updates. The accepted upstream recovery limitation is recorded above; it is not evidence of a complete #161 fix.

Item	Recommended disposition	Evidence and launch acceptance
Portal #241	Merged; deployed-candidate acceptance pending	UID-fenced canonical Pod/Workload/Ray ownership and Event correlation, read RBAC, and unknown/unavailable handling are merged. The final correction preserves same-incarnation troubleshooting evidence when ownership sources are incomplete. Confirm those behaviors on the combined deployed candidate rather than the separate older demo build.
Metrics watch #232	Merged; deployed-candidate acceptance pending	Non-scalar chunk survival, checkpoint persistence, later scalar export, completion, and strict malformed/non-finite input handling are merged. Confirm the integrated release's collection and completion path; an older demo is not qualification of the merged fix.
#161 / #247	Automatic terminal-operation recovery deferred; migration gate remains	#247 merged native state migration and bounded HTTP-error retry hardening. The owner accepted the unsupported asynchronous terminal-operation retry as a documented non-blocking v0.4.2 limitation, not a complete fix. Still require a real refreshed existing-deployment plan proving the grant is preserved and candidate-specific recorder/history acceptance where enabled. Keep #161 open.
#162 / #265	Waiter safety fix required	Fault injection on cdcd1df8 reproduced false readiness with required Functions missing and deletion of a foreign throttled Function in both waiter engines. #265 implements required-set and ownership safeguards and is still undergoing CI/review. Direct Helm bypasses the mitigation and upstream Azure/adx-mon#1238 remains open. Healthy live Functions or a successful waiter do not establish upstream automatic reconciliation.
#190	Recommend defer the upstream completion contract	Main no longer waits on the missing condition and uses skipvalidation, avoiding the known install wait failure. That does not prove ADX schemas are ready; early workload history can be missed. Before first qualification workload, establish schema readiness separately. Upstream Azure/adx-mon#983 remains unmerged; do not promise lossless startup capture.
#111	Recommend defer, with unsupported-field clarification	scratchMount exists only in the CLI model/fixture; there is no production consumer and no CRD property. Hand-authored values cannot persist, but this is not an active implemented mount feature regressing. Do not advertise workspace scratch defaults; remove/clarify the unused field and misleading docs in follow-up, or implement the full schema-to-rendering contract. Explicit Job storage is available, not a universal RayJob scratch workaround.
#76	Recommend defer only with retry restrictions	Fresh invocations generate new submission IDs; durable request-ID/digest replay is absent. Stable authored workload names prevent ordinary same-name duplicate creation, and same-process uncertain-create recovery exists. Renamed/deleted/cross-kind retries can still execute again. Require callers to preserve namespace/kind/name and inspect uncertain results; do not promise cross-invocation idempotency or allow blind automated retries. Block any release feature requiring that guarantee. Open #31 is not an idempotency fix.

Candidate integration cautions and evidence

• Fix portal Job detail metadata matching #241 was merged after exact-head approval and completed checks; no known code-integration blocker remains. Live acceptance must still confirm UID-fenced ownership and preserved diagnostics on the combined candidate.
• fix(portal): keep metrics watch alive without scalars #232 was merged after completed exact-head checks; its old dependency-download failure is not a current blocker. Live acceptance must still cover the combined importer/watch path.
• fix(terraform): retry lifecycle recorder ADX assignment #247 is merged with a native moved block and saved-plan guard, but the real refreshed migration plan remains unproven. Its earlier successful principal creation followed by Helm readiness timeouts is not full installation acceptance. The accepted terminal-operation recovery limitation does not waive migration safety.
• adx-mon: retry Function reconciliation after transient ADX throttling #162 mitigation lives in terraform/aks/wait-for-adx-functions-ready.sh and terraform/aks/Wait-ForAdxFunctionsReady.ps1. Fence ADX Function waiter readiness and retries to the intended release #265 addresses the reproduced required-set/ownership defects; do not run an unfixed waiter against unrelated resources or mismatched release values.
• [Bug]: WorkspaceDefaults.scratchMount is not in the TauWorkspace CRD and is silently pruned #111 evidence spans the CLI workspace type, controller WorkspaceDefaults type, and generated/chart CRDs. Make project submission idempotent by client request ID #76 evidence is in cli/internal/cli/run.go and run_submit.go.

5. Release and deployment gates

• Assign release/Portal acceptance owners, bug owners, target environments, and rollout window; explicitly accept or reject remaining proposed bug deferrals. Only the upstream asynchronous recovery limitation and the three enhancements have accepted deferrals.
• Freeze the release commit and reconcile the complete v0.4.0-to-candidate change list. All tracked enhancements are excluded in favor of 0.4.3; other open PRs are not implicitly included.
• Confirm unified 0.4.2 charts, images, CLI defaults, examples, and release notes; preserve independently versioned SDK, gpu-monitoring, and adx-mon contracts.
• Complete candidate CI and issue-specific regression coverage, including Helm 3/4, CRD retention, and relevant CLI/Portal/controller/SDK contracts.
• Establish a reversible installation/upgrade and ownership transition, with safe Terraform grant migration if fix(terraform): retry lifecycle recorder ADX assignment #247 is included; identify the actual prior deployed revision.
• Resolve or explain and bound the previously reported intermittent Portal HTTP 400; lack of fresh recurrence alone is not a root cause.
• Publish approved v0.4.2 artifacts through release workflows and record versions/digests/workflow links.
• Deploy through existing ownership mechanisms; record environment, release commit, chart version, image digests, and time.
• Verify schema/controller/monitoring readiness before representative workloads, then workspace/queue routing, Job/RayJob execution/logs, serving/checkpoints, and Portal workflows.
• Record go/no-go, roll out, and observe health/error signals for an agreed period.
• Close only after deployment evidence, Portal acceptance, and every bug's fix or accepted deferral are recorded. Refresh the issue inventory at freeze.

Alternatives considered

• Retain v0.4.1: rejected because main already prepares v0.4.2.
• Require every enhancement before launch: rejected; all three are explicitly moved to 0.4.3.
• Close bugs merely because a workaround or PR exists: rejected; distinguish mitigation, code integration, and deployment acceptance.

Additional context

This is release coordination, not an instruction to deploy, publish, merge, grant permissions, or close linked bugs. The triage leaves source unchanged. No live cluster or Azure operations were performed. GitHub Copilot assisted with issue triage and this tracker update.

Confirmation

• I searched existing issues and roadmap items for this request.
• This request does not include confidential or security-sensitive information.

[chokevin]

Blocker execution assigned - 2026-09-09 PDT

Three focused execution sessions are now assigned to prepare existing PRs for merge:

PR / issue	Execution scope	Required outcome
#241	Reused Job detail metadata session	Integrate current main, fix canonical discovery and exact ownership correlation, preserve diagnostic/source-recovery behavior, resolve conflicts and review feedback.
#232	Metrics watch release blocker PR session	Demonstrate metadata-only chunks no longer terminate watch; preserve checkpoints, later scalar export, completion, and failure handling.
#247 / #161	ADX installation release blocker PR session	Resolve unsafe resource/state migration, establish bounded targeted retries, and demonstrate supported existing-install/fresh-install behavior. Also assess the related #162 installation-path gate without claiming the upstream bug is fixed.

Each owner must return the exact PR head, meaningful behavior evidence, current review/CI state, and explicit remaining acceptance gaps. The coordinator will review readiness before merge. This kickoff does not authorize merging, publication, deployment, live resource changes, or Terraform state mutation. Enhancements #113, #92 and #83 remain deferred to 0.4.3.

AI assistance: GitHub Copilot is coordinating these implementation sessions.

[chokevin]

Portal blocker integration update

• fix(portal): keep metrics watch alive without scalars #232: current-main integration and focused fixes are pushed at 05ab112a0269d89cc7c4643707b577b5fd763e42. Exact-head GitHub status is CLEAN/MERGEABLE; 35 checks completed (32 success, 3 release-only skipped). Offline watcher/importer/restart/checkpoint/completion coverage and independent review are recorded in the PR. The recorded human approval covers the older 2ecf5395 revision, not this head: renewed review/merge decision remains outstanding.
• Fix portal Job detail metadata matching #241: current-main integration is pushed at cf2b69b507ec9213ab0b6f2cd8845c2e6c52c61a. Exact-head GitHub status is CLEAN; 42 checks completed (39 success, 3 skipped). The owner reported successful offline ownership/diagnostics/API/UI/RBAC coverage and independent review. Final current-head review disposition is being consolidated.
• fix(terraform): retry lifecycle recorder ADX assignment #247/Terraform: retry lifecycle recorder ADX principal assignment after managed identity propagation #161: still in progress. Provider-native state migration, actual retry failure-shape coverage, and existing-state no-destroy evidence remain the substantive gates; a plan guard or static configuration check is not a substitute for that evidence.

These are code-integration updates, not release-candidate live acceptance. No PR was merged, no bug was closed, and no deployment or publication was performed.

[chokevin]

Blocker disposition update - 2026-09-09, 21:42 PDT

Work	Current disposition
#232 at 05ab112a0269d89cc7c4643707b577b5fd763e42	OPEN/CLEAN; awaiting renewed maintainer review/merge decision. No merge or live acceptance claimed.
#241 at cf2b69b507ec9213ab0b6f2cd8845c2e6c52c61a	New CHANGES_REQUESTED/BLOCKED disposition since the prior handoff. Existing owner has been asked to address the new review, not start a duplicate session.
#247 at 7943c5f950087dec16f09316d2b08f81eaaef631	Native AzureRM-to-AzAPI state migration and a fail-closed saved-plan guard are now implemented; current CI completed with 28 success/3 skipped. Still CHANGES_REQUESTED/BLOCKED and NOT merge-ready.

Remaining ADX evidence gates:

1. An authorized, normally refreshed plan against a named existing deployment/backend/workspace and unchanged inputs must establish identity/body preservation and no destructive grant change. The provider move RPC and plan guard are not substitutes. Use an approved identity, disable automatic provider registration, keep plan/state artifacts private, and do not apply/import/mutate state.
2. The original Terraform: retry lifecycle recorder ADX principal assignment after managed identity propagation #161 report lacks enough HTTP/poll detail to establish retry coverage. AzAPI handles the targeted HTTP errors but does not resubmit a PUT for an HTTP-200 terminal failed LRO. Obtain sanitized original response context or separately authorize isolated reproduction; do not claim complete repair from narrowed documentation.
3. Recorder readiness, actual schema readiness, and durable Portal history still need approved release qualification; initial principal creation followed by Helm timeout is not full acceptance.

#162 remains a conditional gate: both Function waiters select namespace-wide resources without ownership/expected-set fencing. The default supported PowerShell 7 path is distinct from the optional Bash path, which requires modern Bash. Direct Helm bypasses the Terraform workaround; upstream automatic reconciliation is still unfixed. No unsafe waiter invocation or live deployment was authorized.

No PRs merged, bugs closed, resources/state modified, or artifacts published by this coordination update.

[chokevin]

Portal Job detail fix ready for merge decision

#241 is now OPEN, CLEAN, and MERGEABLE at c8e4cef5772e8a4d806d57305c84e334bb2aa4e6. The latest correction preserves troubleshooting evidence by making Events unavailable when Pod or RayCluster ownership sources cannot provide a complete identity set.

The coordinator confirmed an APPROVED review by feiskyer explicitly on this final head, not merely an older approval or an empty reviewDecision. All 42 checks completed (39 successful, 3 expected skipped); the owner reports zero unresolved review threads.

This clears the PR code/review gates for a maintainer merge decision. It does not establish release-candidate live acceptance. No merge, deployment, or publication was performed.

[chokevin]

Post-merge release status - 2026-09-10, 00:07 PDT

• fix(portal): keep metrics watch alive without scalars #232 merged at 2ed3c292e5d6d7446e33ba45ca961f67f09909cb: metrics watch fix is on main.
• fix(terraform): retry lifecycle recorder ADX assignment #247 merged at 72b68ce6e0e601768530cb4ee3ab0d1f69b87434: ADX retry/migration hardening is on main, but Terraform: retry lifecycle recorder ADX principal assignment after managed identity propagation #161 remains open. The PR still records missing actual refreshed existing-state plan, unresolved original HTTP-vs-terminal-LRO error shape, and full-install acceptance. Merge is not evidence those gates are met.
• Fix portal Job detail metadata matching #241 remains open, CLEAN/MERGEABLE at c8e4cef5772e8a4d806d57305c84e334bb2aa4e6, with explicit approval on that head. It is the remaining Portal blocker PR to land.

Next: land #241, select the combined release candidate, and complete approved candidate-specific installation/upgrade plus Portal/workload acceptance, including the unresolved dashboard HTTP400 and #162 supported-installation path. Record explicit dispositions for remaining bugs before go/no-go. Enhancements remain assigned to 0.4.3.

v0.4.0 is still the latest published GitHub release. No release publication, merge, or deployment was performed by this coordination update.

[chokevin]

All three blocker PRs are merged

At the release owner's request, #241 was squash-merged into main at cdcd1df87d16ce8e554baf8b3f6ce79ba064e058 (2026-09-10 00:09 PDT), using an exact-head guard on approved revision c8e4cef5772e8a4d806d57305c84e334bb2aa4e6 and the repository's allowed squash policy.

#232, #247, and #241 are now merged. Code integration is complete for these three PRs; this does not close the unresolved ADX migration/retry/full-install evidence gates or certify the combined release candidate. #161 remains open. Release-candidate acceptance, remaining bug dispositions, and publication are still outstanding. No release tag or deployment was performed.

[chokevin]

Live rehearsal progress - release gate remains open

The demo owner reports a completed clean two-runtime reset and native Helm installation on the separate demo revision 89306b10a9a0d2b107d67d4d9f836950305258be. The runtimes received new identities and reached Ready; recovery was prepared and no rollback was needed. This is meaningful installation evidence, but it is not qualification of combined release main cdcd1df8.

The single fresh GPU rehearsal is still running. Its first ten observed metrics matched real Kusto data; timing was measured with a late observer and does not establish ingestion latency or a 60-second visibility guarantee. Final run retrieval, CPU viewer, and actual UI acceptance remain pending. A narrowly scoped UI read-permission gap is being addressed under separate authorization in the demo session.

The original intermittent HTTP400 cause remains unknown. No HTTP errors in one observation window and the separate reproduced error-classification fix do not establish its resolution. No Terraform/ADX acceptance or evidence for the combined #241/#232/#247 candidate was produced by this rehearsal.

The release remains on hold pending candidate-specific acceptance, the outstanding ADX evidence/disposition, and release-note completion.

[chokevin]

ADX qualification result: working telemetry, unresolved initialization/recovery gates

Offline tests used exact combined candidate cdcd1df87d16ce8e554baf8b3f6ce79ba064e058; live checks were explicitly authorized, bounded, read-only Flex observations. No live infrastructure, Functions, grants, Terraform state, or workloads were changed.

Observed working: all nine live Function CRs reported current-generation success; GpuHealth executed successfully against recent GPU data. The completed demo retained experiment metrics and a status marker. An initial zero-row query used the wrong namespace predicate; the corrected deployed workspace/project/run label mapping returned the retained data. This is not a data-loss or general ADX outage finding.

Confirmed candidate gaps:

1. Both Terraform Function waiters can declare readiness when required Functions are absent (including GpuHealth), and can delete a foreign throttled Function even when all required owned Functions are healthy. Twenty-two isolated adverse cases reproduced sixteen safe outcomes and six unsafe outcomes, three per script engine. All deletions were mocked; no live fault injection occurred. This strengthens the adx-mon: retry Function reconciliation after transient ADX throttling #162 installation gate from a source concern to reproduced behavior.
2. The actual configured AzAPI 2.12 client/SDK retries initial matching AAD HTTP errors and 429s, and promptly rejects nonmatching permanent errors. But an HTTP-200 terminal failed long-running operation containing the AAD error performs no new PUT. Persistent polling failure loses the final AAD diagnostic; Retry-After above the configured delay cap fails visibly instead of recovering. Production retry/timeout values were exercised with virtual time, not actual Azure latency or a whole Terraform operation. The original Terraform: retry lifecycle recorder ADX principal assignment after managed identity propagation #161 response shape is still unknown.
3. The real refreshed existing-deployment migration plan remains untested. Synthetic mover RPC and mock plan/guard coverage do not replace it.

Live evidence boundaries: Flex does not provision the dedicated lifecycle-history table/function and recorder capability, so retained experiment metrics do not establish durable lifecycle UID/completion history. Its deployed Portal/controller also differ from the combined release candidate.

Recommendation: do not cut the full advertised release yet. Fix the narrowly scoped waiter required-set/ownership defects; establish supported identity-failure behavior and upgrade evidence; then qualify the combined candidate or explicitly narrow the supported release claims. No production fixes or broader live authorization were implied by this test pass.

[chokevin]

Separate demo workflow completed; release-candidate acceptance still open

The demo owner reports the completed path on Portal source 89306b10: clean runtime reinstall, one fresh GPU training run, serving its exact new checkpoint in a CPU-only viewer, authenticated browser interaction, and Portal discovery/optimizer charts for that run. The viewer used no GPUs; no second learner or shared ADX change was made. A separately authorized narrowly scoped Portal read permission corrected its profile banner.

This is useful end-to-end demo evidence, but it does not qualify combined release candidate cdcd1df8. The original HTTP400 root cause remains unknown; the 60-second visibility gate was not established, and an approval pause means this was not an uninterrupted timed rehearsal. Dedicated lifecycle-history acceptance remains outstanding.

The reproduced Function waiter ownership/required-set defects and the ADX identity/migration gates from the preceding qualification report remain unchanged. The demo completion does not waive those release gates.

[chokevin]

ADX fixes authorized and assigned

The release owner authorized code fixes following the reproduced qualification failures.

• Function waiter safety: a fresh issue session is implementing required-Function-set and release-ownership enforcement in both PowerShell and Bash, guarded retry targets, consistent failure behavior, and durable regressions for the six reproduced unsafe outcomes. A new focused PR will relate to adx-mon: retry Function reconciliation after transient ADX throttling #162 without claiming the upstream controller retry issue is fully fixed.
• Identity recovery: the existing ADX owner is evaluating a supported released provider/configuration remedy using the actual transport reproductions. Any concrete implementation will go on a fresh follow-up branch, not the merged fix(terraform): retry lifecycle recorder ADX assignment #247 branch. Provider forks, imperative identity/state workarounds, and broader upstream changes require a separate decision.

This authorizes source fixes and offline evidence, not live infrastructure/Function/permission changes, Terraform state or migration-plan operations, deployment, merge, or release publication. The current healthy Flex telemetry and separate demo do not waive the remaining candidate acceptance gates.

[chokevin]

Explicit release-scope decision

The release owner approved treating ONLY the upstream AzAPI asynchronous terminal-operation retry gap as a documented, non-blocking v0.4.2 limitation. The issue body now records this decision, and release-note PR #266 carries the user-facing wording.

#161 remains open. This is a scope deferral, not a fix or proof of the original incident's response shape. Function waiter safety (#265), actual permission-migration evidence, and combined-candidate acceptance remain required. No upstream provider fork/change, cap adjustment, or live operation was authorized by this decision.

[chokevin]

Documentation PR closed; scope decision unchanged

#266 was closed unmerged at the release owner's request. The existing terraform/aks/README.md already documents the HTTP-only retry behavior, terminal-operation limitation, and operator recovery guidance; another release-note PR is not needed for recording the scope decision.

The narrowly accepted asynchronous-recovery deferral remains in this tracker; #161 remains open. It must not be presented as a proven cause of the original incident, whose full HTTP/poll response was not captured. This does not waive the separately reproduced waiter safety defects addressed by #265 or any applicable existing-deployment migration requirement.

[chokevin]

Waiter fix: implementation complete, renewed review requested

#265 is at a1169f4caf1ddde37e7a986b6cae8eebbbf328ab. The reviewer's DNS-name validation finding has been corrected: Function names use Kubernetes DNS-subdomain limits while namespaces retain DNS-label limits. The sole review thread is answered and resolved. All current checks are complete (32 successful, 3 expected skipped).

GitHub still records CHANGES_REQUESTED from the older head. Renewed review was requested from feiskyer; no further code defect is currently identified in the review. The PR remains unmerged pending that decision.

#266 stays closed. The upstream asynchronous-retry limitation remains deferred; no additional provider investigation, documentation PR, live operation, merge, or release publication was started.