fix: safely clean up Fabric test artifacts after 24 hours by ranadeepsingh · Pull Request #2728 · microsoft/SynapseML

Rana Singh (ranadeepsingh) · 2026-09-18T23:30:55Z

Related Issues/PRs

Separate infrastructure fix for Fabric E2E failures caused by workspace artifact
quota exhaustion.

What changes are proposed in this pull request?

Replace the existing name-only, three-day cleanup with a strict 24-hour policy
for artifacts owned by this repository's Fabric tests.

Require both creation and last-update timestamps to be older than 24 hours in
UTC. Tag new test artifacts with an explicit OSS ownership description and
conservatively recognize legacy names and descriptions.
Retain unrelated, ambiguous, active, scheduled, recently changed, or shared
items. Re-read inventory before deleting; keep stores whenever job ownership
or consumer relationships do not establish safe deletion.
Delete job definitions before stores. Check absence up to 31 times with
two-second pauses instead of assuming a successful DELETE is immediately
visible. A concurrent not-found response still requires confirmed absence.
Attempt independent job deletions after individual failures, retain stores,
and report collected failures. Let Fabric cascade managed SQL endpoints
rather than deleting endpoints independently.
Add SYNAPSEML_FABRIC_CLEANUP_DRY_RUN=true, operator documentation, and safety
regressions in an existing CI-selected suite.
Make cleanup a named Fabric cleanup preflight task inside the existing
FabricE2E job, after authentication/build setup and before the success-gated
E2E task. Preserve cleanup JUnit results before the next SBT invocation and
publish staged reports and phase metadata even when E2E is skipped or fails.
Defer suite Fabric connections, stores, and notebook submissions until test
execution. Each suite independently caches its cleanup and setup outcomes,
including interrupted cleanup. Failed setup cannot retry allocations for
later tests. Successful runs retain concurrency 3, immediate per-job cleanup,
and executor shutdown before final artifact cleanup.

This changes OSS test infrastructure and the Fabric pipeline task layout.
Public main APIs, generated code, workflows, dependency pins, and other
repositories are unchanged.

How is this patch tested?

Reproduced eager workspace resolution and missing preflight task failures
before implementing the guards and pipeline split.
JDK 11: 43 tests passed across FabricTestArtifactTrackerSuite and
FabricArtifactNamesSuite, with no skipped tests.
Full compile and Test/compile.
All-module scalastyle and Test/scalastyle.
86 pipeline regressions, including execution of the actual Bash snippets
with fake commands for success, cleanup failure, E2E failure, and failed setup.
Black 22.3.0: 207 files unchanged.
Six review rounds for the initial cleanup and six for this preflight
follow-up. Findings and resolutions are committed under reviews/pr-2728/.
Live preview and authorized execute pass of the cleanup policy before the
content-preserving rebase and preflight orchestration follow-up.

Both live passes examined 24 items, found 0 owned jobs and 0 owned
stores, and confirmed 0 deletions. No unrelated items were touched, and
these passes did not reclaim capacity.

The live passes did not exercise the real deletion/confirmation, job-history,
or schedule paths because no items were eligible. Deterministic tests cover
those safeguards. Endpoint shapes and sparkjob were checked against the official
history,
schedule,
and Spark job type
contracts.

The new preflight orchestration has not yet completed a live Fabric CI run.
Local checks and a queued Azure build do not establish that the original
Fabric E2E failures are resolved.

Does this PR change any dependencies?

No.

Does this PR add a new feature? If so, have you added samples on website?

No public feature. Cleanup usage and preflight behavior are documented in
docs/Reference/Developer Setup.md.

azure-pipelines · 2026-09-18T23:31:05Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

github-actions · 2026-09-18T23:31:05Z

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

fix: Fix LightGBM crashes with empty partitions
feat: Make HTTP on Spark back-offs configurable
docs: Update Spark Serving usage
build: Add codecov support
perf: improve LightGBM memory usage
refactor: make python code generation rely on classes
style: Remove nulls from CNTKModel
test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

Rana Singh (ranadeepsingh) · 2026-09-18T23:31:20Z

/azp run

azure-pipelines · 2026-09-18T23:31:34Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot review overview

🔵 Needs a closer look

Destructive infrastructure cleanup paths require final human review, and validation records contain unresolved inconsistencies.

Review effort: Lite
Findings: None

What changed in this PR

Updates Fabric test artifact cleanup to safely remove only owned, stale artifacts with dependency checks, dry-run support, and deletion confirmation.

Changes:

Adds conservative 24-hour ownership and activity filtering.
Adds guarded deletion, pagination, failure aggregation, and regression tests.
Documents cleanup operation and records review validation.

File	Summary
`reviews/task-fabric-cleanup-attempt-1-review-6-claude-opus-5.md`	Final review and validation record.
`reviews/task-fabric-cleanup-attempt-1-review-5-gemini-3.8-flash.md`	Testing review and resolutions.
`reviews/task-fabric-cleanup-attempt-1-review-4-gpt-6-astra.md`	Failure-aggregation review record.
`reviews/task-fabric-cleanup-attempt-1-review-3-claude-opus-5.md`	Edge-case findings and resolutions.
`reviews/task-fabric-cleanup-attempt-1-review-2-gemini-3.8-flash.md`	Architecture review record.
`reviews/task-fabric-cleanup-attempt-1-review-1-gpt-6-astra.md`	Initial review findings.
`docs/Reference/Developer Setup.md`	Documents cleanup usage and safeguards.
`core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricTestArtifactTrackerSuite.scala`	Adds cleanup safety regressions.
`core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricNotebookTests.scala`	Integrates cleanup and dry-run handling.
`core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricArtifactCleanup.scala`	Implements guarded cleanup logic.
`core/src/test/scala/com/microsoft/azure/synapse/ml/fabric/FabricOperations.scala`	Integrates Fabric APIs and ownership metadata.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

## Summary Replace name-only three-day cleanup with ownership-aware, UTC-based 24-hour retention for Fabric E2E artifacts. Confirm asynchronous deletion before reclaiming stores, and add deterministic safety regressions and operator docs. ## Prompting Intent Investigate failing Fabric E2E pipelines, delete workspace items created by this repository's tests only when older than 24 hours, and publish the change as a separate PR. Preserve unrelated, active, shared, and ambiguous items. ## Linked Sources - Existing cleanup and suite selection: https://github.com/microsoft/SynapseML/blob/cd45147c7025f483e86fc028069d72b070e73a55/core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricNotebookTests.scala - Existing Fabric adapter and artifact creation: https://github.com/microsoft/SynapseML/blob/cd45147c7025f483e86fc028069d72b070e73a55/core/src/test/scala/com/microsoft/azure/synapse/ml/fabric/FabricOperations.scala - Operator documentation: docs/Reference/Developer Setup.md - Six sequential review artifacts: reviews/task-fabric-cleanup-attempt-1-review-*.md - Job history contract: https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/list-item-job-instances - Schedule contract: https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/list-item-schedules - Spark job type: https://learn.microsoft.com/en-us/rest/api/fabric/sparkjobdefinition/background-jobs/run-on-demand-spark-job-definition ## Rationale Artifact names alone do not establish safe ownership or activity. Require matching ownership metadata, expired creation/update timestamps, safe dependencies, and idle unscheduled jobs. Re-read inventory before writes; poll deletion visibility rather than repeat DELETE. Retain stores on any unconfirmed job deletion, and report independent failures explicitly. Keep the change in test infrastructure without altering public APIs or CI configuration. This infrastructure bug fix carries the area/build PR label. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-09-19T00:14:37Z

/azp run

azure-pipelines · 2026-09-19T00:14:43Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

Rana Singh (ranadeepsingh) · 2026-09-19T00:15:40Z

/azp run

azure-pipelines · 2026-09-19T00:15:53Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot review overview

🔵 Needs a closer look

Fabric job-history and schedule requests use a Power BI token audience instead of the Fabric API audience, which can block cleanup.

Review effort: Lite
Findings: None

AB#5628913 ## Summary Move the six existing Fabric cleanup review artifacts into reviews/pr-2728/ without changing their contents or the reviewed implementation. ## Prompting Intent The engineer requested reviews/pr-<pr_number>/ for review artifacts and an assigned ADO work item in the current sprint linked to this pull request. ## Linked Sources - Pull request: microsoft#2728 - Work item: https://dev.azure.com/msdata/A365/_workitems/edit/5628913 - Existing review rounds: reviews/pr-2728/task-fabric-cleanup-attempt-1-review-*.md ## Rationale Scope the audit trail to its PR while preserving all original review findings, resolutions, and evidence. Git blob checks confirm six content-identical moves. No source code, dependencies, or CI configuration changes are needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-09-19T00:25:14Z

/azp run

azure-pipelines · 2026-09-19T00:25:27Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot review overview

🟡 Changes recommended

Relation parsing can silently ignore malformed dependency fields and permit unsafe deletion.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

Reject malformed relation IDs instead of silently dropping dependency edges · New

AB#5628913 ## Summary Separate the existing Fabric cleanup from E2E execution into a named, fail-closed pipeline task after authentication and build setup. Preserve cleanup reports across the second SBT invocation and publish staged evidence on failure. Defer Fabric workspace lookup, store allocation and notebook submission until test execution. Cache per-suite cleanup and setup outcomes, preserve interrupts, and retain bounded parallel notebook execution and tracked artifact cleanup. ## Prompting Intent The engineer asked to make repository-owned test-item cleanup a pre-job task before Fabric E2E rather than hiding it inside the notebook task. They approved the explicit pipeline preflight plus lazy suite-level guards. Retain the strict 24-hour ownership policy in the separate cleanup PR, link their current-sprint ADO task, and keep all review artifacts under reviews/pr-2728/. ## Linked Sources - Pull request: microsoft#2728 - Tracking task: https://dev.azure.com/msdata/A365/_workitems/edit/5628913 - Operator guide: docs/Reference/Developer Setup.md - Review findings and resolutions: reviews/pr-2728/task-5628913-attempt-2-review-*.md ## Rationale A separate step exposes cleanup failures without introducing another job or duplicating setup. Per-suite checks protect direct runs independently of CI. Cached setup failures prevent later tests from retrying allocations. Laziness allows test registration without connecting to Fabric while preserving the existing parallel workload and public suite member types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-09-19T02:19:21Z

/azp run

azure-pipelines · 2026-09-19T02:19:33Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot review overview

🔵 Needs a closer look

Resolve the notebook setup-error handling and cleanup initialization metadata issues, and correct the documented test command.

Review effort: Lite
Findings: 1

Open (1)

Reject malformed relation IDs instead of silently dropping dependency edges

AB#5628913 ## Summary Make NotebookFixture abstract and create anonymous concrete instances where its owning suite executes it. Preserve the repository-wide CI test coverage gate. ## Prompting Intent The engineer reported failed Azure jobs on the Fabric cleanup PR and supplied the test-matrix and TLS errors. Investigate each failed job, repair the introduced fixture regression, and distinguish unrelated infrastructure and port prerequisites. ## Linked Sources - PR: microsoft#2728 - Failed build: https://dev.azure.com/msdata/A365/_build/results?buildId=236598594 - Task: https://dev.azure.com/msdata/A365/_workitems/edit/5628913 - Port prerequisite: microsoft#2725 - Review: reviews/pr-2728/task-5628913-attempt-3-review-1-gpt-6-astra.md ## Rationale The private helper is exercised by its parent suite, not a standalone test entry point. Making that intent explicit fixes source discovery without adding a fake CI selector or weakening the guard. Preserve TLS validation and strict port replay; those failures have separate infrastructure and branch-content causes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-09-19T05:52:58Z

CI failure triage for build 236598594:

UnitTests core: fixed in 16c74e638fee4f7cc1b6fb1051a0977c262d8587. The new nested NotebookFixture was incorrectly treated as a standalone suite. It is now abstract, with anonymous instances created by its owning suite. The exact failing coverage gate was reproduced before the correction and passes afterward; all 44 targeted tests, Scala style, and compile/Test compile pass. No scanner exclusion or pipeline selector was added.
UnitTests language: the AzureCLI task failed before SBT started. The connection to msdata.visualstudio.com received a certificate for *.azureedge.net. This is an agent/service TLS failure, not a language-test assertion. Certificate verification remains enabled.
Spark 4.1 compatibility: replay conflicts in FabricNotebookTests.scala. Current spark4.1 tip b4ca894139 lacks the immediate-artifact cleanup methods introduced on master by fix: harden CI cleanup and retain reusable branch guidance #2725, commit 1f33e37653. That prerequisite must reach the release branch before this replay can pass. Retrying the unchanged port cannot resolve the conflict; the gate has not been weakened.

Evidence and the direct review are in reviews/pr-2728/task-5628913-attempt-3-review-1-gpt-6-astra.md. Fresh validation will check the core fix and retry infrastructure setup, but the release prerequisite remains an explicit blocker.

Rana Singh (ranadeepsingh) · 2026-09-19T05:53:18Z

/azp run

azure-pipelines · 2026-09-19T05:53:31Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Copilot review overview

🔵 Needs a closer look

The substantial infrastructure and Fabric CI changes lack a completed live CI validation.

Review effort: Lite
Findings: 1

Open (1)

Reject malformed relation IDs instead of silently dropping dependency edges

Rana Singh (ranadeepsingh) · 2026-09-21T08:40:20Z

/azp run

azure-pipelines · 2026-09-21T08:40:37Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Codecov Comments Bot (codecov-commenter) · 2026-09-21T09:33:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.65%. Comparing base (3878cae) to head (16c74e6).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2728   +/-   ##
=======================================
  Coverage   87.65%   87.65%           
=======================================
  Files         343      343           
  Lines       21176    21176           
  Branches     2232     2232           
=======================================
  Hits        18561    18561           
  Misses       2615     2615

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

AB#5628913 ## Summary Move the six existing Fabric cleanup review artifacts into reviews/pr-2728/ without changing their contents or the reviewed implementation. ## Prompting Intent The engineer requested reviews/pr-<pr_number>/ for review artifacts and an assigned ADO work item in the current sprint linked to this pull request. ## Linked Sources - Pull request: #2728 - Work item: https://dev.azure.com/msdata/A365/_workitems/edit/5628913 - Existing review rounds: reviews/pr-2728/task-fabric-cleanup-attempt-1-review-*.md ## Rationale Scope the audit trail to its PR while preserving all original review findings, resolutions, and evidence. Git blob checks confirm six content-identical moves. No source code, dependencies, or CI configuration changes are needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

AB#5628913 ## Summary Separate the existing Fabric cleanup from E2E execution into a named, fail-closed pipeline task after authentication and build setup. Preserve cleanup reports across the second SBT invocation and publish staged evidence on failure. Defer Fabric workspace lookup, store allocation and notebook submission until test execution. Cache per-suite cleanup and setup outcomes, preserve interrupts, and retain bounded parallel notebook execution and tracked artifact cleanup. ## Prompting Intent The engineer asked to make repository-owned test-item cleanup a pre-job task before Fabric E2E rather than hiding it inside the notebook task. They approved the explicit pipeline preflight plus lazy suite-level guards. Retain the strict 24-hour ownership policy in the separate cleanup PR, link their current-sprint ADO task, and keep all review artifacts under reviews/pr-2728/. ## Linked Sources - Pull request: #2728 - Tracking task: https://dev.azure.com/msdata/A365/_workitems/edit/5628913 - Operator guide: docs/Reference/Developer Setup.md - Review findings and resolutions: reviews/pr-2728/task-5628913-attempt-2-review-*.md ## Rationale A separate step exposes cleanup failures without introducing another job or duplicating setup. Per-suite checks protect direct runs independently of CI. Cached setup failures prevent later tests from retrying allocations. Laziness allows test registration without connecting to Fabric while preserving the existing parallel workload and public suite member types. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Restrict per-artifact cleanup recovery to NonFatal. Interrupts and fatal cleanup errors now escape even when the notebook body has already failed. Preserve the body error as suppressed where supported, with an identity guard. Add 15 throwable/body combinations and a shared-instance regression to the existing CI-selected tracker suite, and document the behavior. ## Prompting Intent The engineer requested separate master sync pull requests for spark4.0 and spark4.1 using the SynapseML PR loop. Address the verified cleanup review finding on master first, then carry the exact commit into both port branches without changing their runtime baselines. ## Linked Sources - Finding: microsoft#2734 (comment) - Master prerequisite: microsoft#2732 - Spark 4.0 sync: microsoft#2733 - Spark 4.1 sync: microsoft#2734 - Original cleanup work: microsoft#2728 - Review records: reviews/fabric-cleanup-relations-20260921/task-fatal-cleanup-attempt-1-review-*.md ## Rationale Use NonFatal for recoverable cleanup and explicitly rethrow excluded errors after retaining the original failure. Keep the body's catch-and-rethrow and ordinary cleanup-error suppression unchanged. Regressions demonstrated both fatal masking and lost body diagnostics before their fixes. The fixed master passes compile, test compile, production/test Scala style, and all 48 tracker and naming tests. Review records retain the unavailable Gemini-family gate; they do not claim complete three-family review or full Azure validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Reject malformed nested relation metadata instead of discarding dependency edges. Preserve earlier deletion failures when a later metadata read aborts, and avoid self-suppression when cleanup reports a reused exception. Add fake-client regressions, document the accepted relation shape, and retain the review findings and their resolutions. ## Prompting Intent The engineer requested separate worktrees and sync pull requests carrying relevant master changes into spark4.0 and spark4.1, using the SynapseML PR loop. Review of that sync exposed these cleanup defects. Isolate portable fixes on master first rather than maintaining different cleanup implementations on ports. ## Linked Sources - microsoft#2728 - microsoft#2728 (comment) - AGENTS.md: cross-version changes land on master before port integration - .github/skills/synapseml-pr-loop/SKILL.md - reviews/fabric-cleanup-relations-20260921/ ## Rationale A partial dependency graph can authorize unsafe deletion, so unknown metadata must abort inventory collection. Keep outer empty/null relation semantics and valid nested GUID references. Preserve the current inventory exception while attaching earlier deletion errors, including failed job-history or schedule reads; never continue deleting after the metadata read fails. Identity guards retain real failures instead of replacing them with a self-suppression exception. This changes test infrastructure only, not public SparkML APIs, dependency pins, pipeline settings, or runtime enablement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Restrict per-artifact cleanup recovery to NonFatal. Interrupts and fatal cleanup errors now escape even when the notebook body has already failed. Preserve the body error as suppressed where supported, with an identity guard. Add 15 throwable/body combinations and a shared-instance regression to the existing CI-selected tracker suite, and document the behavior. ## Prompting Intent The engineer requested separate master sync pull requests for spark4.0 and spark4.1 using the SynapseML PR loop. Address the verified cleanup review finding on master first, then carry the exact commit into both port branches without changing their runtime baselines. ## Linked Sources - Finding: microsoft#2734 (comment) - Master prerequisite: microsoft#2732 - Spark 4.0 sync: microsoft#2733 - Spark 4.1 sync: microsoft#2734 - Original cleanup work: microsoft#2728 - Review records: reviews/fabric-cleanup-relations-20260921/task-fatal-cleanup-attempt-1-review-*.md ## Rationale Use NonFatal for recoverable cleanup and explicitly rethrow excluded errors after retaining the original failure. Keep the body's catch-and-rethrow and ordinary cleanup-error suppression unchanged. Regressions demonstrated both fatal masking and lost body diagnostics before their fixes. The fixed master passes compile, test compile, production/test Scala style, and all 48 tracker and naming tests. Review records retain the unavailable Gemini-family gate; they do not claim complete three-family review or full Azure validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Check deletion immediately, then at most ten more times with 30-second waits per artifact. Preserve the existing fail-closed ordering and error handling. Inject the sleep duration so tests verify 30000 ms and a 300000 ms wait budget. Expand boundary coverage and document that HTTP request time is additional. ## Prompting Intent The engineer explicitly requested 30-second polling with a five-minute maximum, superseding the earlier one-minute request. Match the existing bounded confirmation pattern, keep one deletion call per candidate, and carry the portable master-first fix into both Spark sync pull requests. ## Linked Sources - Engineer request: "update it so it polls every 30 seconds with a max timeout of 5 mins" - Master prerequisite: microsoft#2732 - Original bounded polling: microsoft#2728 - Spark 4.0 sync: microsoft#2733 - Spark 4.1 sync: microsoft#2734 - Review records: reviews/pr-2732/task-cleanup-polling-30s-attempt-1-review-*.md ## Rationale Use eleven reads and ten waits rather than a shared or wall-clock deadline, matching the referenced per-item wait-budget semantics. Duration-aware sleeper injection tests the actual requested delays without real sleeps. Regressions cover immediate, first, middle, and final success, fresh child and parent budgets, exhaustion, inventory errors, conflicting metadata, interrupts, and protected consumers. Five unchanged failure tests move into the existing mix-in to preserve the 800-line limit and CI test registration. The old 31-read/two-second policy failed the new assertions. The final master passes core compile, test compile, production/test Scala style, and all 50 tracker/naming tests. No runtime pins, pipeline policy, or release tooling changes are included. Gemini-family review remains explicitly unavailable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Reject malformed nested relation metadata instead of discarding dependency edges. Preserve earlier deletion failures when a later metadata read aborts, and avoid self-suppression when cleanup reports a reused exception. Add fake-client regressions, document the accepted relation shape, and retain the review findings and their resolutions. ## Prompting Intent The engineer requested separate worktrees and sync pull requests carrying relevant master changes into spark4.0 and spark4.1, using the SynapseML PR loop. Review of that sync exposed these cleanup defects. Isolate portable fixes on master first rather than maintaining different cleanup implementations on ports. ## Linked Sources - #2728 - #2728 (comment) - AGENTS.md: cross-version changes land on master before port integration - .github/skills/synapseml-pr-loop/SKILL.md - reviews/fabric-cleanup-relations-20260921/ ## Rationale A partial dependency graph can authorize unsafe deletion, so unknown metadata must abort inventory collection. Keep outer empty/null relation semantics and valid nested GUID references. Preserve the current inventory exception while attaching earlier deletion errors, including failed job-history or schedule reads; never continue deleting after the metadata read fails. Identity guards retain real failures instead of replacing them with a self-suppression exception. This changes test infrastructure only, not public SparkML APIs, dependency pins, pipeline settings, or runtime enablement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Restrict per-artifact cleanup recovery to NonFatal. Interrupts and fatal cleanup errors now escape even when the notebook body has already failed. Preserve the body error as suppressed where supported, with an identity guard. Add 15 throwable/body combinations and a shared-instance regression to the existing CI-selected tracker suite, and document the behavior. ## Prompting Intent The engineer requested separate master sync pull requests for spark4.0 and spark4.1 using the SynapseML PR loop. Address the verified cleanup review finding on master first, then carry the exact commit into both port branches without changing their runtime baselines. ## Linked Sources - Finding: #2734 (comment) - Master prerequisite: #2732 - Spark 4.0 sync: #2733 - Spark 4.1 sync: #2734 - Original cleanup work: #2728 - Review records: reviews/fabric-cleanup-relations-20260921/task-fatal-cleanup-attempt-1-review-*.md ## Rationale Use NonFatal for recoverable cleanup and explicitly rethrow excluded errors after retaining the original failure. Keep the body's catch-and-rethrow and ordinary cleanup-error suppression unchanged. Regressions demonstrated both fatal masking and lost body diagnostics before their fixes. The fixed master passes compile, test compile, production/test Scala style, and all 48 tracker and naming tests. Review records retain the unavailable Gemini-family gate; they do not claim complete three-family review or full Azure validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Check deletion immediately, then at most ten more times with 30-second waits per artifact. Preserve the existing fail-closed ordering and error handling. Inject the sleep duration so tests verify 30000 ms and a 300000 ms wait budget. Expand boundary coverage and document that HTTP request time is additional. ## Prompting Intent The engineer explicitly requested 30-second polling with a five-minute maximum, superseding the earlier one-minute request. Match the existing bounded confirmation pattern, keep one deletion call per candidate, and carry the portable master-first fix into both Spark sync pull requests. ## Linked Sources - Engineer request: "update it so it polls every 30 seconds with a max timeout of 5 mins" - Master prerequisite: #2732 - Original bounded polling: #2728 - Spark 4.0 sync: #2733 - Spark 4.1 sync: #2734 - Review records: reviews/pr-2732/task-cleanup-polling-30s-attempt-1-review-*.md ## Rationale Use eleven reads and ten waits rather than a shared or wall-clock deadline, matching the referenced per-item wait-budget semantics. Duration-aware sleeper injection tests the actual requested delays without real sleeps. Regressions cover immediate, first, middle, and final success, fresh child and parent budgets, exhaustion, inventory errors, conflicting metadata, interrupts, and protected consumers. Five unchanged failure tests move into the existing mix-in to preserve the 800-line limit and CI test registration. The old 31-read/two-second policy failed the new assertions. The final master passes core compile, test compile, production/test Scala style, and all 50 tracker/naming tests. No runtime pins, pipeline policy, or release tooling changes are included. Gemini-family review remains explicitly unavailable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) requested a review from Brendan Walsh (BrendanWalsh) as a code owner September 18, 2026 23:30

Copilot AI lite review requested due to automatic review settings September 18, 2026 23:30

Rana Singh (ranadeepsingh) added bug area/build labels Sep 18, 2026

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) September 18, 2026 23:31 View session

Copilot AI reviewed Sep 18, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 19, 2026 00:14

Rana Singh (ranadeepsingh) force-pushed the fix/fabric-test-cleanup-24h-20260918 branch from a016ddb to 1205df2 Compare September 19, 2026 00:14

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) September 19, 2026 00:15 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 19, 2026 00:24

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) September 19, 2026 00:25 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes

Comment thread core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricArtifactCleanup.scala

Copilot AI review requested due to automatic review settings September 19, 2026 02:19

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) September 19, 2026 02:19 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 19, 2026 05:52

Copilot started reviewing on behalf of Rana Singh (ranadeepsingh) September 19, 2026 05:53 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes

Rana Singh (ranadeepsingh) merged commit 37321d0 into microsoft:master Sep 21, 2026
77 of 78 checks passed

This was referenced Sep 21, 2026

fix: preserve Fabric cleanup safety and error propagation #2732

Merged

chore: bring latest master fixes to Spark 4.0 #2733

Merged

chore: bring latest master fixes to Spark 4.1 #2734

Merged

Conversation

Rana Singh (ranadeepsingh) commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issues/PRs

What changes are proposed in this pull request?

How is this patch tested?

Does this PR change any dependencies?

Does this PR add a new feature? If so, have you added samples on website?

Uh oh!

azure-pipelines Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 18, 2026

Uh oh!

azure-pipelines Bot commented Sep 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 19, 2026

Uh oh!

azure-pipelines Bot commented Sep 19, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 19, 2026

Uh oh!

azure-pipelines Bot commented Sep 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 19, 2026

Uh oh!

azure-pipelines Bot commented Sep 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 19, 2026

Uh oh!

azure-pipelines Bot commented Sep 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 19, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 19, 2026

Uh oh!

azure-pipelines Bot commented Sep 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 21, 2026

Uh oh!

azure-pipelines Bot commented Sep 21, 2026

Uh oh!

Codecov Comments Bot (codecov-commenter) commented Sep 21, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Rana Singh (ranadeepsingh) commented Sep 18, 2026 •

edited

Loading