Skip to content

CI: AgentToolTrackingControllerTests delivery-barrier test can drop observer event on shard 2 #389

Description

@baron

Impact

CI app shard 2 can fail on an Agent Mode tool-tracking concurrency test, blocking otherwise unrelated PR validation noise and making it hard to distinguish product regressions from test instability.

Observed behavior

On PR #388, exact head a1b5fbe8, Build and Test (app shard 2) failed twice on the same test after a one-time rerun:

  • AgentToolTrackingControllerTests/testConcurrentRawUnregisterAndStopJoinCapturedDeliveryBarrier
  • Failure location: Tests/RepoPromptTests/AgentMode/AgentToolTrackingControllerTests.swift:227
  • Assertion: expected firedCount == 1, observed firedCount == 0
  • Distinctive log excerpt:
[ClaudeToolTracking] MCP observer call skipped no event observers

The failed workflow was the PR CI run for head a1b5fbe8; all other CI jobs for that head completed successfully.

Expected behavior

The test should deterministically capture the intended delivery-barrier event, or the contract should be rewritten so the ordering it asserts is guaranteed by the implementation.

Evidence suggesting this is unrelated to PR #388

PR #388 changes MCP agent-run/worktree inheritance and startup-pending wait snapshot behavior. It does not touch:

  • Tests/RepoPromptTests/AgentMode/AgentToolTrackingControllerTests.swift
  • AgentToolTrackingController
  • the tool-tracking observer registration/unregistration implementation path implicated by the log excerpt

A local focused run of the failing test passed:

make dev-test FILTER='AgentToolTrackingControllerTests/testConcurrentRawUnregisterAndStopJoinCapturedDeliveryBarrier'

That combination points to a timing/order-sensitive CI flake or latent concurrency issue rather than a regression in the PR's changed code path.

Reproduction notes

  1. Run PR CI app shard 2 on a head that includes the current test suite.
  2. Observe whether AgentToolTrackingControllerTests/testConcurrentRawUnregisterAndStopJoinCapturedDeliveryBarrier fails with firedCount == 0.
  3. Compare against a focused local run of the same test.

Frequency from the observed PR: 2/2 hosted shard-2 attempts failed after the initial failure and one rerun; focused local run passed once.

Acceptance criteria

  • The test no longer fails because observer delivery sees an empty observer map under the captured-delivery-barrier scenario, or the test is adjusted to assert only behavior guaranteed by the production contract.
  • A focused run of AgentToolTrackingControllerTests/testConcurrentRawUnregisterAndStopJoinCapturedDeliveryBarrier passes.
  • CI shard 2 can pass without requiring repeated reruns for this specific test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:agent-modeAgent Mode runtime and interfacebugSomething isn't workingprovider:claudeClaude-compatible provider behavior

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions