Harden test harness against hangs and waiter leaks#435
Conversation
|
My hero |
# Conflicts: # Tests/RepoPromptTests/MCP/Control/MCPSharedServerTestLease.swift # Tests/RepoPromptTests/MCP/Control/PersistentAgentModeMCPReadFileConnectionTests.swift # Tests/RepoPromptTests/WorkspaceContext/Search/SearchPathFilteringTests.swift
baron
left a comment
There was a problem hiding this comment.
Approved for merge after local review, Fable second-opinion review, follow-up hardening fixes, focused local validation, and green exact-head CI.
|
Follow-up on the CI failure: The failing The issue was that tool-event observers are stored by unregisterToolObservers(for: runID)That is correct for true end-of-run cleanup, but too broad for controller-owned teardown. If another observer was registered for the same The fix changes owner-scoped teardown to be token-scoped:
So the test hardening/sharding did not create this lifecycle issue; it made the race visible by exercising tighter async cleanup timing. |
baron
left a comment
There was a problem hiding this comment.
Thanks — agreed this is the better fix. Token-scoped teardown addresses the root cause: controller-owned cleanup was run-wide scoped, so it could remove observers it didn’t register.
I walked the concurrency with Fable: observer delivery leases are acquired synchronously before the fire path’s first suspension, so token removal plus waitUntilIdle() preserves the drain guarantee. The fallback join on an in-flight run-wide unregister also looks right.
I don’t see a blocker in this approach. Current CI failures are now in different suites, so I’d treat those as the remaining blockers to investigate/rerun rather than continuing to debate token-scoped vs test-only cleanup.
|
Debugging the CI now. |
|
@morluto I'll stay out of your way and let you handle this just tag me here or there when you want me to look at it. Will just merge it if solid with no pushback for pushback's sake. Thank you! |
|
CI is clean, LGTM @baron |
baron
left a comment
There was a problem hiding this comment.
Re-approving current head. The token-scoped observer teardown approach looks sound, and CI is green on this revision.
|
@morluto tysm for your help! |
Summary
This PR hardens the test harness so missed cancellation, release, socket-read, or subprocess-drain paths fail with bounded assertions instead of hanging the suite.
What was wrong before
Several test helpers could park indefinitely or rely on weak cancellation behavior:
nil/falsesilently, making timeouts easy to under-assertThese failures are expensive because they usually show up as stuck test runs rather than direct XCTest failures.
What changed
TestProcessRunner/TestGitCommandRunnerwith process-group cleanup, runner timeouts, and output-drain timeoutsReview order
Tests/RepoPromptTests/Helpers/TestHangHardenedFences.swiftTests/RepoPromptTests/Helpers/TestProcessRunner.swiftOut of scope
This PR intentionally does not include CI sharding, broad assertion cleanup, oversized test-file consolidation, or ledger/scoreboard work.
Validation
make dev-formatmake dev-lintmake dev-testCodemapBindingEngineInvalidationTestsCodemapBindingEngineRootLeaseTestsFileSystemContentLoadingConcurrencyTestsWindowCloseCoordinatorLifecycleTests