Skip to content

Prepare hosted app tests for optional sharding#440

Open
morluto wants to merge 4 commits into
repoprompt:mainfrom
morluto:test-runner-sharding-prep
Open

Prepare hosted app tests for optional sharding#440
morluto wants to merge 4 commits into
repoprompt:mainfrom
morluto:test-runner-sharding-prep

Conversation

@morluto

@morluto morluto commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the runner mechanics and suite metadata needed to canary process-level sharding for hosted app tests. Default CI behavior remains serial because --workers defaults to 1.

What Changed

  • Added Scripts/Fixtures/ci-test-serial-groups.json for shared-resource serial lanes.
  • Extended Scripts/ci_app_test_runner.py with:
    • --ledger
    • --serial-group-policy
    • --workers
    • --print-suite-plan-json
  • Aggregates contract-ledger rows into suite-level tags, runtime estimates, and execution tiers.
  • Classifies suites as pinned serial or parallel eligible.
  • Keeps pinned suites out of the worker pool when --workers > 1.
  • Stops in-flight suite subprocesses through the existing process cleanup path after a failure.
  • Adds runner self-tests for policy parsing, suite planning, serial ordering, worker scheduling, fail-fast behavior, and missing config errors.
  • Adds targeted teardown for known shared-state suites.
  • Updates ledger tags for touched suites without changing scenario counts or method IDs.

Review Notes

Suggested review order:

  1. Scripts/ci_app_test_runner.py
  2. Scripts/test_ci_app_test_runner.py
  3. Scripts/Fixtures/ci-test-serial-groups.json
  4. Scripts/Fixtures/test-suite-contract-ledger.tsv
  5. Targeted XCTest teardown changes under Tests/RepoPromptTests

The main behavior to verify is that --workers 1 preserves current serial execution, while --workers > 1 schedules only parallel-eligible suites in the worker pool.

Validation

Passed:

  • make ci-app-test-runner-selftest
  • python3 -m py_compile Scripts/ci_app_test_runner.py Scripts/test_ci_app_test_runner.py
  • git diff --check
  • make dev-test-shard-plan SHARDS=4
  • .agents/skills/rpce-contribution-check/scripts/preflight.sh push
  • .agents/skills/rpce-contribution-check/scripts/preflight.sh pr-ready

Scope

This PR does not enable parallel hosted CI. It only adds the config, runner behavior, metadata, and targeted teardown needed for a later canary.

@morluto morluto changed the title Prepare app test runner for optional sharding Prepare hosted app tests for optional sharding Jul 9, 2026
…g-prep

# Conflicts:
#	Scripts/Fixtures/test-suite-contract-ledger.tsv
#	Scripts/ci_app_test_runner.py
#	Scripts/test_ci_app_test_runner.py
#	Tests/RepoPromptTests/MCP/MCPReadSearchLatencyDiagnosticsGuardTests.swift
#	Tests/RepoPromptTests/WorkspaceContext/WorkspaceFileContextStoreTests.swift
@morluto morluto marked this pull request as ready for review July 9, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant