[Issue] hipSPARSE quick conversion tests fail on Windows-gfx110X with SEH exception 0xc0000005

[jayhawk-commits]

Description:

hipSPARSE quick conversion tests are failing in TheRock CI on Windows gfx110X-all with SEH exception 0xc0000005.

Failures noticed in CI of rocm-libraries bump PR: ROCm/TheRock#5387
Commit history: e3b6f33...d5bf18f

Context:

• Platform: Windows
• Impacted Arch: gfx110X-all
• Run: https://github.com/ROCm/TheRock/actions/runs/26228464899
• Job: Windows::release / Test gfx110X-all / Test hipsparse / Test hipsparse (shard 1/1) (gfx110X-all)

Full log:

• https://github.com/ROCm/TheRock/actions/runs/26228464899/job/77419812085

Failed Tests:

quick/prune_csr2csr_by_percentage.conversion/f64_r_500_623_0_0b_0b
quick/prune_csr2csr_by_percentage.conversion/f64_r_500_623_50_0b_0b
quick/prune_csr2csr_by_percentage.conversion/f64_r_500_623_100_0b_0b
quick/prune_csr2csr_by_percentage.conversion/f64_r_872_33_0_0b_0b
quick/prune_csr2csr_by_percentage.conversion/f64_r_872_33_50_0b_0b
quick/prune_csr2csr_by_percentage.conversion/f64_r_872_33_100_0b_0b
quick/coo2csr.conversion/f32_c_0_0_0b_0
quick/coo2csr.conversion/f32_c_0_0_1b_0
quick/coo2csr.conversion/f32_c_0_33_0b_0
quick/coo2csr.conversion/f32_c_0_33_1b_0
quick/csrsort.conversion/f32_r_0_1000_0_1b
quick/csrsort.conversion/f32_r_0_1000_1_1b
quick/csrsort.conversion/f32_r_10_0_0_0b

Test Summary:

./build/bin/hipsparse-test '--gtest_filter=*quick*:-known_bug*:*csr2csr_compress*:*prune_csr2csr.conversion*'

unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.

subprocess.CalledProcessError: Command '['./build/bin/hipsparse-test', '--gtest_filter=*quick*:-known_bug*:*csr2csr_compress*:*prune_csr2csr.conversion*']' returned non-zero exit status 3221226505.

Impact:

Blocker for promotion of rocm-libraries submodule to TheRock: ROCm/TheRock#5387

[amd-chiranjeevi]

@ntrost57
Could you take a look at this? The issue is intermittent, but it’s occurring too frequently to ignore.
Can we check whether these flaky tests are being handled properly, or consider skipping them?

[kliegeois]

Investigation summary: #7713 is a recurrence of the known, machine-specific gfx110X-Windows crash (TheRock#4533), not a bump regression

TL;DR

• The Test hipsparse … (gfx110X-all) failures correlate with the CI runner, not the bumped code.
• Each failing run is a single GPU fault in prune_csr2csr_by_percentage → device loss → cascade, not many independent test bugs.
• This is the **same failure class as ([Issue] [hipSPARSE][CI][Failure] csr2csr_compress and prune_csr2csr failing (f64) with invalid status and pointer mode errors TheRock#4533), which was reported as machine-specific / not reproducible on any dev machine and only ever worked around ([ci] excluding crashing gfx110X windows tests for hipsparse TheRock#4609), never root-caused.

1. Result tracks the runner, not the build

Sweep of recent bump PRs' gfx110X hipsparse check (4 jobs gh reported as "fail" were actually cancelled and are excluded):

PR	Bump	Runner	Result
5387	libraries e3b6f33→d5bf18f	runner-25	FAIL
5417	libraries 5bd2f90→807283e	runner-07	pass
5427	systems 470c276→ed97534	runner-16	pass
5428	libraries 807283e→a893ef8	runner-00	pass
5437	libraries 807283e→60e5be1	runner-04	pass
5457	libraries 60e5be1→62d3a26	runner-19	pass
5465	systems ed97534→4a30df8	runner-00	pass
5466	libraries 60e5be1→d591a7c	runner-22	FAIL
5483	libraries 62d3a26→46653de	runner-22	FAIL
5492	systems 4a30df8→cb65612	CS-RORDMZ-DT201	pass
5493	libraries 62d3a26→e34ac06	CS-RORDMZ-DT78	FAIL
5506	libraries 62d3a26→0dda48e	CS-RORDMZ-DT78	FAIL
5510	systems cb65612→ab8dfe9	runner-23	FAIL
5511	libraries 46653de→ca4930e	CS-RORDMZ-DT85	pass
5541	systems cb65612→b8c378e	runner-21	FAIL

Repeated runs on the same runner are consistent across different builds:

• runner-00 passes two different builds (5428 libraries, 5465 systems)
• runner-22 fails two different builds (5466, 5483)
• CS-RORDMZ-DT78 fails two different builds (5493, 5506)

The runners partition cleanly with no overlap:

• Always pass: runner-00, 04, 07, 16, 19, DT85, DT201
• Always fail: runner-21, 22, 23, 25, DT78

Because the same runner gives the same verdict across different binaries, the determining
variable is the runner/host environment (GPU driver/firmware/state), not the bumped
source. Bump type is mixed on both sides — both rocm-libraries and rocm-systems
bumps appear in the pass set and the fail set — so this is not a regression in either
submodule.

2. Failure mode: one fault + a device-loss cascade

On the failing runs whose logs are still retained (5387/runner-25, 5510/runner-23, 5541/runner-21):

• The first failure is always a quick/prune_csr2csr_by_percentage.conversion/... case
  (e.g. f64_r_500_623_0, f64_r_872_33_0, f64_r_10_623_0).
• That first test runs slowly (~0.3–1.0 s vs ~50 ms normally), then throws SEH 0xc0000005.
• Every subsequent test then fails in 0–2 ms, and hipMalloc begins returning
  hipErrorOutOfMemory(2) — the HIP context/device is gone.
• Zero numeric/assertion mismatches; this is purely a crash/stability event.

So the large coo2csr / csrsort / identity failure counts are collateral from device
loss, not independent bugs. The real trigger is the prune_csr2csr_by_percentage (rocSPARSE)
path, which faults on the affected runners.

3. This is a recurrence of TheRock#4533 (not new, not a bump regression)

The same gfx110X-Windows hipSPARSE conversion crash was reported in ROCm/TheRock#4533
(2026-04-14) for csr2csr_compress and prune_csr2csr, then worked around in
ROCm/TheRock#4609 (2026-04-16) by excluding those two suites on gfx110X. In #4533 the owner
reported it as machine-specific and not reproducible on any dev machine, with the
identical signature seen here: a slow first failure (~0.4–0.9 s) followed by every
subsequent test failing (device-loss cascade), and a different test failing first on
re-run (nondeterministic). Root cause was never found.

prune_csr2csr_by_percentage (failing in this issue) has existed since 2020 and was simply
not in the #4609 exclusion list, so the same failure class leaks through it. Note the
current CI filter excludes *prune_csr2csr.conversion* but not prune_csr2csr_by_percentage.

Timeline:

Date	Event
2020-09-08	prune_csr2csr_by_percentage test introduced (hipSPARSE #102/#135)
2026-04-14	TheRock#4533: csr2csr_compress + prune_csr2csr crash on gfx110X-Windows; machine-specific; not root-caused
2026-04-16	TheRock#4609: those two muted on gfx110X (workaround)
2026-05-22 → 06-01	#7713: prune_csr2csr_by_percentage (un-muted sibling) trips the same cascade on the same "bad" runners

Caveats

Observational; the key runners have 2 samples each. The one clean confirmation still to do is
running the identical artifact on a known-good vs known-bad runner a few times each.