Skip to content

Commit 2e33f29

Browse files
fix: merge fatal cleanup propagation from master prerequisite
## Summary Merge the exact master-first fatal-cleanup correction and its regression, documentation, and review records into this Spark sync branch. ## Prompting Intent The engineer requested separate spark4.0 and spark4.1 sync worktrees and pull requests with the SynapseML PR loop. Preserve each port's runtime and compatibility differences while carrying portable fixes by normal merge. ## Linked Sources - Finding: microsoft#2734 (comment) - Master prerequisite: microsoft#2732 - Spark 4.0 sync: microsoft#2733 - Spark 4.1 sync: microsoft#2734 - Review records: reviews/fabric-cleanup-relations-20260921/task-fatal-cleanup-attempt-1-review-*.md ## Rationale Retain shared source and real merge ancestry rather than duplicate the fix or alter branch-specific settings. The master regression demonstrated the fatal-error masking bug before the NonFatal correction. This port passes compile, test compile, production/test Scala style, and 48 tracker/naming tests. Full Azure validation and the unavailable Gemini-family review gate remain explicit; this commit makes no merge-readiness claim. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2 parents ef20129 + 81ccc54 commit 2e33f29

9 files changed

Lines changed: 475 additions & 1 deletion

‎core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricTestArtifactTracker.scala‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,15 @@ private[nbtest] final class FabricTestArtifactTracker(deleteArtifact: String =>
2929
deleteTrackedArtifact(artifactId)
3030
artifactIds.remove(artifactId)
3131
} catch {
32-
case cleanupError: Throwable =>
32+
case NonFatal(cleanupError) =>
3333
failure match {
3434
case Some(original) =>
3535
if (original ne cleanupError) original.addSuppressed(cleanupError)
3636
case None => throw cleanupError
3737
}
38+
case cleanupError: Throwable =>
39+
failure.filterNot(_ eq cleanupError).foreach(cleanupError.addSuppressed)
40+
throw cleanupError
3841
}
3942
}
4043
}

‎core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricTestArtifactTrackerFailureTests.scala‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,55 @@ package com.microsoft.azure.synapse.ml.nbtest
66
import org.scalatest.funsuite.AnyFunSuite
77

88
import scala.collection.mutable.ArrayBuffer
9+
import scala.util.control.ControlThrowable
910

1011
private[nbtest] trait FabricTestArtifactTrackerFailureTests extends AnyFunSuite {
12+
test("Propagate fatal per-artifact cleanup errors after successful or failed work") {
13+
val cleanupFailures = Seq[() => Throwable](
14+
() => new InterruptedException("cleanup interrupted"),
15+
() => new InternalError("cleanup VM failure"),
16+
() => new ThreadDeath(),
17+
() => new LinkageError("cleanup linkage failure"),
18+
() => new ControlThrowable {})
19+
cleanupFailures.foreach { newCleanupFailure =>
20+
Seq[Option[Throwable]](None, Some(new IllegalStateException("job failed")),
21+
Some(new InternalError("job VM failure"))).foreach { jobFailure =>
22+
val cleanupFailure = newCleanupFailure()
23+
var attempts = 0
24+
val tracker = new FabricTestArtifactTracker(_ => {
25+
attempts += 1
26+
if (attempts == 1) throw cleanupFailure
27+
})
28+
val thrown = intercept[Throwable] {
29+
tracker.withArtifact("job") { _ =>
30+
jobFailure.foreach(throw _)
31+
"completed"
32+
}
33+
}
34+
assert(thrown eq cleanupFailure)
35+
val suppressionProbe = newCleanupFailure()
36+
suppressionProbe.addSuppressed(new RuntimeException("suppression probe"))
37+
val expectedSuppressed = if (suppressionProbe.getSuppressed.isEmpty) Seq.empty else jobFailure.toSeq
38+
assert(thrown.getSuppressed.toSeq == expectedSuppressed)
39+
assert(attempts == 1)
40+
tracker.cleanup()
41+
assert(attempts == 2)
42+
tracker.cleanup()
43+
assert(attempts == 2)
44+
}
45+
}
46+
}
47+
48+
test("Preserve a repeated fatal cleanup throwable without self-suppression") {
49+
val failure = new InterruptedException("cleanup interrupted")
50+
val tracker = new FabricTestArtifactTracker(_ => throw failure)
51+
val thrown = intercept[InterruptedException] {
52+
tracker.withArtifact("job") { _ => throw failure }
53+
}
54+
assert(thrown eq failure)
55+
assert(thrown.getSuppressed.isEmpty)
56+
}
57+
1158
test("Attempt all deletions and preserve cleanup failures") {
1259
val attempted = ArrayBuffer.empty[String]
1360
val firstFailure = new RuntimeException("first failure")

‎docs/Reference/Developer Setup.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,10 @@ cleanup or starting notebook work; successful notebook runs retain their bounded
8686
parallel execution and per-job artifact cleanup. Interrupted cleanup preserves
8787
the interrupt signal. Store creation and executor setup failures are also
8888
cached, so later tests do not repeat initialization or start another notebook batch.
89+
Per-job cleanup never suppresses an interrupt or fatal error behind a notebook
90+
failure. The cleanup throwable escapes with the earlier notebook failure attached
91+
where that throwable permits suppression. An unsuccessful per-job deletion stays
92+
tracked for final cleanup.
8993

9094
Set `SYNAPSEML_FABRIC_CLEANUP_DRY_RUN=true` to preview eligible deletions without
9195
changing the workspace. Omit it, or set it to `false`, to perform cleanup.
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
## Review summary
2+
3+
- Round: 1, broad sweep only
4+
- Theme: Correctness, security, logic, and conformance to the requested cleanup behavior
5+
- Mode: sequential
6+
- Model: gpt-6-astra
7+
- Target: master follow-up for microsoft/SynapseML#2734, comment 4066689707
8+
- Base HEAD: `02272e0a5d6f986a09289149310f5475c2457b1b`
9+
- Scope: The uncommitted three-file delta and relevant tracker context
10+
- Artifact: `reviews\fabric-cleanup-relations-20260921\task-fatal-cleanup-attempt-1-review-1-gpt-6-astra.md`
11+
- Issues found: 0
12+
- Verdict: CLEAN
13+
14+
## Reviewed working-tree snapshot
15+
16+
| Repository-relative path | Git blob hash |
17+
| --- | --- |
18+
| `core\src\test\scala\com\microsoft\azure\synapse\ml\nbtest\FabricTestArtifactTracker.scala` | `8ddef631521c4730b20dbd3d9dd93efbc51ecfee` |
19+
| `core\src\test\scala\com\microsoft\azure\synapse\ml\nbtest\FabricTestArtifactTrackerFailureTests.scala` | `717c93f7b504ab7b29eb6bc6d3a10497b0baf895` |
20+
| `docs\Reference\Developer Setup.md` | `9701a07632698d1b0ba7211da0ebc1165bb9436d` |
21+
22+
## Evidence checklist
23+
24+
- [x] Inspected the exact delta against the stated HEAD. The implementation changes only the per-artifact cleanup catch from `Throwable` to `NonFatal`; the body catch and rethrow are unchanged.
25+
- [x] Traced successful work, ordinary body failure, and fatal body failure. An excluded cleanup throwable escapes the `finally` block unchanged rather than becoming suppressed behind the body error.
26+
- [x] Checked ordinary cleanup behavior remains intact: preserve a prior body error with distinct cleanup errors suppressed, or throw the cleanup error when the body succeeded. The self-suppression guard is unchanged.
27+
- [x] Checked artifact retention. Cleanup failure exits before deque removal, permitting final cleanup to retry; successful cleanup removes the entry and prevents another deletion.
28+
- [x] Reviewed all 15 regression combinations: `InterruptedException`, `InternalError` as a `VirtualMachineError`, `ThreadDeath`, `LinkageError`, and `ControlThrowable`, each with successful, ordinarily failed, and fatally failed work.
29+
- [x] The regression uses fresh cleanup throwable instances, asserts exact identity and attempt counts, retries the retained artifact, then checks that a second final cleanup performs no duplicate deletion.
30+
- [x] Confirmed the private trait is mixed into the CI-selected `FabricTestArtifactTrackerSuite`. The new case appears under that concrete suite in the green log.
31+
- [x] Checked documentation matches the exception precedence and retained-artifact behavior. No public signature, network destination, credential handling, or serialization changes are introduced.
32+
- [x] Inspected `master-fatal-cleanup-red.log`: the new test fails on old code because the body `IllegalStateException` escapes instead of the same cleanup `InterruptedException`.
33+
- [x] Inspected completed `master-fatal-cleanup-green.log`: JDK 11 core compile and Test/compile succeeded; main and test scalastyle report zero errors and warnings; 47 tests passed across two completed suites, with zero failures, canceled, ignored, pending, or aborted cases.
34+
- [x] The focused working-tree diff passes whitespace checking. No scoped changes were staged during this review.
35+
36+
## Conclusion and boundary
37+
38+
No concrete defect found in this bounded delta. The fix restores propagation of interruption, fatal JVM errors, and control throwables without changing ordinary failure handling or artifact retry bookkeeping.
39+
40+
Validation claims above come from inspected parent-produced logs, not reviewer-run builds. Only this report was written; no source edits, agents, staging, commits, pushes, or remote calls were performed. No new port validation is claimed.
41+
42+
The strict Gemini-family gate remains unavailable following confirmed backend 400 failures. This is a Round 1 result only, not a full gauntlet pass or an overall readiness declaration.
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Fatal cleanup follow-up, round 2
2+
3+
## Review summary
4+
5+
- Theme: architecture and patterns.
6+
- Model: GPT-6 Astra, parent review in the unavailable Gemini slot.
7+
- Scope: the three-file fatal-cleanup follow-up to `02272e0a5d`.
8+
- Issues found: 0.
9+
- Verdict: CLEAN for this bounded review, not a full gauntlet pass.
10+
11+
## Evidence
12+
13+
- `core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricTestArtifactTracker.scala`
14+
narrows only the cleanup recovery handler to `NonFatal`. This matches bulk
15+
cleanup and `FabricNotebookTests.shutdownAndCleanup`. The body handler still
16+
records and immediately rethrows every throwable so ordinary cleanup failures
17+
cannot mask a fatal body failure.
18+
- No public signature, serialized parameter, runtime setting, dependency, or
19+
pipeline definition changes. The helper remains private test infrastructure.
20+
- `FabricTestArtifactTrackerFailureTests.scala` follows the existing private
21+
test-trait structure. `FabricTestArtifactTrackerSuite` already mixes it in, so
22+
the existing explicit pipeline selector includes the new regression.
23+
- `docs/Reference/Developer Setup.md` documents fatal propagation and retained
24+
artifact tracking without claiming that later cleanup is guaranteed to run.
25+
- Local compile, production/test Scala style, and the tracker plus naming suites
26+
passed with 47 tests. Evidence is retained locally in
27+
`master-fatal-cleanup-green.log`, not published as a repository artifact.
28+
29+
## Model limitation
30+
31+
The Gemini slot was not executed for this follow-up. Earlier session attempts
32+
with the available Gemini models failed at the backend with HTTP 400 before
33+
review execution. This fallback does not satisfy independent Gemini coverage.
Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
# Review 3 — Edge Cases and Robustness
2+
3+
- **Task:** `task-fatal-cleanup`, attempt 1 | **Round:** 3 | **Model:** claude-opus-5
4+
- **Branch:** master sync worktree, delta relative to HEAD `02272e0a5d`
5+
- **Trigger:** Copilot finding on microsoft/SynapseML#2734, comment 4066689707
6+
7+
## Scope
8+
9+
The three-file uncommitted follow-up only:
10+
`core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/FabricTestArtifactTracker.scala`
11+
(one catch clause), `.../FabricTestArtifactTrackerFailureTests.scala` (+33), and
12+
`docs/Reference/Developer Setup.md` (+2). No wider sync or history re-audit.
13+
14+
## Verdict
15+
16+
**ISSUES_FOUND — 1 Low.** The reported bug is genuinely fixed and the regression is a
17+
real one. The Low is the mirror-image information loss the narrowed catch introduces.
18+
19+
## Priority matrix after the change
20+
21+
"Excluded" means anything `NonFatal` rejects: interrupts, `VirtualMachineError`,
22+
`ThreadDeath`, `LinkageError`, `ControlThrowable`.
23+
24+
| body outcome | cleanup outcome | result |
25+
| --- | --- | --- |
26+
| success | success | body value returned |
27+
| success | NonFatal | cleanup error thrown |
28+
| success | excluded | cleanup throwable thrown |
29+
| any failure | NonFatal | body error thrown, cleanup attached as suppressed |
30+
| any failure | excluded | cleanup throwable thrown, **body error discarded** |
31+
32+
The last row is correct on priority — the interrupt or fatal now wins, which is the
33+
point of the fix — but the body error is not merely deprioritised. Because the throw
34+
leaves a `finally`, the in-flight body exception is replaced outright: it is neither
35+
rethrown nor attached, so nothing records it.
36+
37+
## L1 (Low) — body failure vanishes when cleanup throws an excluded throwable
38+
39+
`withArtifact` still assigns `failure = Some(error)` in the body catch, but on the
40+
excluded-throwable path nothing reads it. The single production caller is
41+
`FabricNotebookTests.withTrackedArtifact`, where the body error is the notebook
42+
failure and the cleanup error is an artifact DELETE. A pipeline timeout that
43+
interrupts the DELETE after a notebook has already failed therefore surfaces only
44+
the `InterruptedException`, and the notebook failure that triggered triage is gone.
45+
46+
This is the same class of defect the earlier rounds of this work fixed, and the
47+
repository already has the idiom for it: rethrow the winner with the loser attached
48+
under an `ne` guard, exactly as `FabricArtifactCleanup.run` and this method's own
49+
`NonFatal` branch do. A second catch clause preserves the new priority while keeping
50+
the body error visible, and the 15-case regression asserts only
51+
`thrown eq cleanupFailure`, so it would stay green:
52+
53+
```scala
54+
case cleanupError: Throwable =>
55+
failure.filterNot(_ eq cleanupError).foreach(cleanupError.addSuppressed)
56+
throw cleanupError
57+
```
58+
59+
If this is instead accepted as a deliberate trade-off, the new documentation sentence
60+
should say so, because "never suppresses an interrupt or fatal error behind a notebook
61+
failure" does not tell a reader that the notebook failure is then dropped.
62+
63+
## Verified correct
64+
65+
- The narrowed catch is the only production edit. `NonFatal` was already imported and
66+
used by `cleanup()`, so no import churn and no unused symbol.
67+
- Identity is exact in all 15 combinations: `assert(thrown eq cleanupFailure)` pins the
68+
same instance rather than the type, which is what the red log disproved.
69+
- The `original ne cleanupError` self-suppression guard on the `NonFatal` path is
70+
untouched, so a shared instance is still never suppressed into itself.
71+
- Tracking and retry: `artifactIds.remove` sits after `deleteTrackedArtifact`, so any
72+
throwing cleanup leaves the id tracked. The test proves the retry precisely —
73+
`attempts == 1` after `withArtifact`, `2` after the first `cleanup()`, and `2` again
74+
after a second `cleanup()`, so the deque drains and nothing is deleted twice.
75+
- `deleteTrackedArtifact` still treats `PowerBIEntityNotFound` as success, so a
76+
concurrently deleted artifact is untracked rather than retried.
77+
- Body fatals are still recorded and rethrown by `case error: Throwable`, so a fatal
78+
body error keeps priority over a `NonFatal` cleanup error.
79+
- Both documentation sentences are accurate: cleanup no longer hides an interrupt or
80+
fatal, and an unsuccessful per-job deletion does stay tracked.
81+
82+
## Evidence
83+
84+
- `master-fatal-cleanup-red.log`: the new test alone, 1 run, 0 succeeded, 1 failed,
85+
with `IllegalStateException: job failed was not the same instance as
86+
java.lang.InterruptedException: cleanup interrupted`. A true red on old behaviour.
87+
- `master-fatal-cleanup-green.log`: `core/compile`, `core/Test/compile`, main and test
88+
scalastyle each `Found 0 errors`, and 47 of 47 tests passing across
89+
`FabricTestArtifactTrackerSuite` and `FabricArtifactNamesSuite`, with the new case
90+
listed by name. 47 is the expected 46 plus one.
91+
- `FabricTestArtifactTrackerFailureTests.scala` is 76 lines and
92+
`FabricTestArtifactTrackerSuite.scala` is unchanged at 790.
93+
94+
## Non-blocking
95+
96+
Tracking the same artifact id through `withArtifact` twice would leave a duplicate in
97+
the deque, since `ConcurrentLinkedDeque.remove` drops one occurrence. That predates
98+
this delta and is unaffected by it; noted only because retry behaviour was in scope.
99+
100+
## Coverage limitations
101+
102+
Round 1 and the parent's round 2 fallback were both GPT-family and found no concrete
103+
issues. No Gemini version has executed at any point in this task — every attempt has
104+
returned a backend HTTP 400 with zero execution — so the three-family review gate is
105+
**unfulfilled** and this is not a full-gauntlet green. This report covers round 3 only.
106+
Azure Pipelines and current-head GitHub review have not run against this delta, and the
107+
ports do not carry it yet, so no port proof is claimed.
108+
109+
## L1 resolution
110+
111+
The parent accepted L1. A separate excluded-throwable handler now attaches the
112+
earlier body error under an identity guard and immediately rethrows the cleanup
113+
throwable. This preserves fatal priority and diagnostics without wrapping it.
114+
The documentation states that attachment depends on throwable suppression support.
115+
116+
The strengthened matrix failed before this handler was added:
117+
`master-fatal-cleanup-diagnostics-red.log` reports an empty suppressed list instead
118+
of the original `IllegalStateException`. The matrix now checks suppressed errors
119+
as well as identity. A fresh throwable probes suppression support independently,
120+
because `ControlThrowable` support differs across the supported Scala versions.
121+
A separate shared-instance interrupt case guards against self-suppression.
122+
Updated green validation and reviewer verification follow below.
123+
124+
## Round 3 verification of the L1 resolution
125+
126+
Verified against the current worktree; the original verdict and history above stand
127+
unchanged. **L1 is resolved. No new issue.**
128+
129+
- **Catch order is safe.** `case NonFatal(cleanupError)` precedes
130+
`case cleanupError: Throwable`, so the broad clause cannot shadow the narrow one and
131+
only excluded throwables reach it. Reversing them would have silently restored the
132+
original bug.
133+
- **Priority is unchanged; only diagnostics changed.** The excluded path still ends in
134+
`throw cleanupError`, so the interrupt or fatal still wins and still replaces the body
135+
error as the propagating exception. The only difference is that the body error is now
136+
attached instead of vanishing, which is exactly the narrow repair L1 asked for.
137+
- **Identity guard is correct in both directions.**
138+
`failure.filterNot(_ eq cleanupError)` yields `None` both when the body succeeded and
139+
when the body threw the same instance, so `addSuppressed` is never called on self and
140+
cannot raise `IllegalArgumentException`.
141+
- **The probe cannot contaminate the assertion.** `newCleanupFailure()` builds a fresh
142+
instance, so `thrown.getSuppressed` is read from an object the probe never touched.
143+
Deriving `expectedSuppressed` from `jobFailure.toSeq` also keeps the body-success rows
144+
expecting an empty list, and the probe makes the matrix portable across the Scala
145+
versions that differ on `ControlThrowable` suppression rather than hard-coding either.
146+
- **The same-instance test is a real negative control.** Without the `filterNot` guard
147+
the tracker would call `addSuppressed` on self and fail with `IllegalArgumentException`
148+
rather than returning the shared `InterruptedException` with an empty suppressed list.
149+
- **Docs are accurate.** All three sentences hold: the cleanup throwable escapes, the
150+
earlier notebook failure is attached, and the qualifier "where that throwable permits
151+
suppression" honestly covers categories that disable suppression instead of
152+
over-promising.
153+
- **Evidence.** `master-fatal-cleanup-diagnostics-red.log` is a true red for the new
154+
assertion — `Array() did not equal List(java.lang.IllegalStateException: job failed)`,
155+
1 run, 0 succeeded, 1 failed. `master-fatal-cleanup-green-v2.log` has completed:
156+
`core/compile`, `core/Test/compile`, main and test scalastyle each `Found 0 errors`,
157+
`Run completed`, and 48 of 48 tests passing across both suites, with both new cases
158+
listed by name. 48 is the expected 47 plus the same-instance test. The log carries no
159+
`[error]` line and no `*** FAILED ***`; its nine keyword hits are test names and the
160+
tracker's own expected `println` diagnostics. All three edited files were last written
161+
before that run, so the log describes this source.
162+
163+
Round 3 verification only. No Gemini version has executed at any point in this task, so
164+
the three-family gate remains **unfulfilled** and this is not a full-gauntlet green.

0 commit comments

Comments
 (0)