feat(bin): add base-head test suite comparison

[coreldh]

Intent

Publish Suite R2 only as exactly one upstream PR. Make the base/head suite partition runner sound while preserving isolation, checkout guard, and per-side inventories: rerun first-only candidate failures so flakes are not findings; classify outcome transitions with a regressed bucket including timeout-to-failure; independently reconcile git-tree enumeration so accounted can refuse; report compared coverage honestly; distinguish genuine fixed-to-passed from deleted/skipped erosion; record in-flight scripts as running/in flight; convert pre-flight errors to exit 2; clean up and write partial partition state on SIGINT/SIGTERM. Preserve the red-before-green firing cases for flaky failure, timeout-to-failure regression, enumeration mismatch, and deleted/skipped erosion. Do not include Routing R2, do not merge, and stop after the PR step completes before approving CI.

What Changed

• Add fm-test-run.sh --compare-commits to run bounded, isolated base/head test inventories and emit per-side plus partition JSON evidence.
• Recheck candidate failures and classify confirmed regressions, flakes, genuine fixes, coverage erosion, and unreconciled inventories; preserve in-flight partial state on interruption.
• Document the comparison mode and add focused regression coverage for retry, timeout, inventory, coverage, preflight, and signal-handling cases.

Risk Assessment

⚠️ Medium: The large stateful comparison runner is source-consistent after the two repairs, but its bounded execution and signal paths remain inherently higher-risk pending the pipeline-owned tests.

Testing

Confirmed target commit 26d0eac with a clean worktree, ran the focused comparison-runner cases only, and retained end-to-end CLI/JSON artifacts showing each required red-before-green behavior plus SIGTERM recovery; all exercised checks behaved as required.

Evidence: Focused comparison-test transcript

$ FM_TEST_RUN_ONLY=test_compare_commits_partitions_and_bounds_every_script bash tests/fm-test-run.test.sh
ok - commit comparison bounds a deliberate hang and emits complete diffable inventories
$ FM_TEST_RUN_ONLY=test_compare_commits_accounts_for_script_lost_after_discovery bash tests/fm-test-run.test.sh
ok - a script lost after discovery remains inventoried as errored and checkout drift fails loudly
$ FM_TEST_RUN_ONLY=test_compare_commits_rechecks_a_first_only_failure bash tests/fm-test-run.test.sh
ok - a first-only failure is rechecked and recorded as flaky, not introduced
$ FM_TEST_RUN_ONLY=test_compare_commits_refuses_an_inconclusive_retry bash tests/fm-test-run.test.sh
ok - an inconclusive retry remains a regression
$ FM_TEST_RUN_ONLY=test_compare_commits_reports_timeout_to_failure_as_regressed bash tests/fm-test-run.test.sh
ok - timeout-to-failure is a confirmed regression and exits non-zero
$ FM_TEST_RUN_ONLY=test_compare_commits_never_calls_deleted_or_skipped_tests_fixed bash tests/fm-test-run.test.sh
ok - only failure-to-pass is fixed; deletion and skip are coverage erosion
$ FM_TEST_RUN_ONLY=test_compare_commits_refuses_an_independent_enumeration_mismatch bash tests/fm-test-run.test.sh
ok - tracked test paths missing from discovery make accounted refuse
$ FM_TEST_RUN_ONLY=test_compare_commits_headline_exposes_measured_coverage bash tests/fm-test-run.test.sh
ok - headline distinguishes timeouts from one genuinely compared outcome
$ FM_TEST_RUN_ONLY=test_compare_commits_preflight_errors_exit_two_without_tracebacks bash tests/fm-test-run.test.sh
ok - bad refs and output paths are infrastructure errors with exact reasons
$ FM_TEST_RUN_ONLY=test_compare_commits_signal_records_inflight_and_cleans_clones bash tests/fm-test-run.test.sh
ok - SIGTERM and SIGINT preserve partial evidence and remove isolated clones
$ FM_TEST_RUN_ONLY=test_compare_commits_signal_records_inflight_confirmation bash tests/fm-test-run.test.sh
ok - SIGTERM during confirmation preserves a running observation

Evidence: End-to-end CLI transcript: flake, regression, erosion, enumeration

CLI partition evidence
summary= {"compared_outcomes": 1, "comparison_total": 4, "coverage": "1/4", "fixed": 0, "flaky": 1, "head_introduced": 0, "inherited": 0, "inherited_timeouts": 0, "no_longer_measured": 2, "now_passing": 0, "regressed": 1}
regressed= [{"base_observations": ["timed_out", "timed_out"], "base_outcome": "timed_out", "head_observations": ["failed", "failed"], "head_outcome": "failed", "path": "tests/ab-timeout.test.sh"}]
flaky_transitions= [{"base_observations": ["passed", "passed"], "base_outcome": "passed", "head_observations": ["failed", "passed"], "head_outcome": "failed", "path": "tests/aa-flaky.test.sh"}]
fixed= []
no_longer_measured= [{"base_outcome": "failed", "head_outcome": "absent", "path": "tests/ac-deleted.test.sh"}, {"base_outcome": "failed", "head_outcome": "skipped", "path": "tests/ad-skipped.test.sh"}]
Enumeration refusal evidence
inventory_reconciled= False
base_missing_from_discovery= ["tests/zz-ghost.test.sh"]

Evidence: Persisted partition JSON from CLI scenario

{
  "base_commit": "fa8362981851140509bfe490960194b7a11d9fa5",
  "checkout_invariants_ok": true,
  "fixed": [],
  "flaky_transitions": [
    {
      "base_observations": [
        "passed",
        "passed"
      ],
      "base_outcome": "passed",
      "head_observations": [
        "failed",
        "passed"
      ],
      "head_outcome": "failed",
      "path": "tests/aa-flaky.test.sh"
    }
  ],
  "head_commit": "e7ebb8b958353d3c6c2c4ed9997e94443f66e112",
  "head_introduced_failures": [],
  "inherited_failures": [],
  "inherited_timeouts": [],
  "inventory_reconciled": true,
  "no_longer_measured": [
    {
      "base_outcome": "failed",
      "head_outcome": "absent",
      "path": "tests/ac-deleted.test.sh"
    },
    {
      "base_outcome": "failed",
      "head_outcome": "skipped",
      "path": "tests/ad-skipped.test.sh"
    }
  ],
  "now_passing": [],
  "regressed": [
    {
      "base_observations": [
        "timed_out",
        "timed_out"
      ],
      "base_outcome": "timed_out",
      "head_observations": [
        "failed",
        "failed"
      ],
      "head_outcome": "failed",
      "path": "tests/ab-timeout.test.sh"
    }
  ],
  "schema_version": 2,
  "summary": {
    "compared_outcomes": 1,
    "comparison_total": 4,
    "coverage": "1/4",
    "fixed": 0,
    "flaky": 1,
    "head_introduced": 0,
    "inherited": 0,
    "inherited_timeouts": 0,
    "no_longer_measured": 2,
    "now_passing": 0,
    "regressed": 1
  }
}

Evidence: SIGTERM partial-state and cleanup evidence

exit_code=143
in_flight_row= {"detail": "in flight", "outcome": "running"}
termination= {"complete": false, "exit_code": 143, "signal": "SIGTERM"}
clone_path_removed= True

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (2) ✅

• 🚨 bin/fm-test-run.sh:1542 - Required criterion: “record in-flight scripts as running/in flight.” A SIGINT/SIGTERM during a retry confirmation reaches this spawned process without updating any inventory row; the signal handler then writes the original terminal outcome (for example failed or timed_out) and a partition with no confirmation observation. Preserve a running/in-flight confirmation state before waiting, so partial artifacts accurately identify the interrupted script.

🔧 Fix: Preserve running confirmation state on interruption
1 error still open:

• 🚨 bin/fm-test-run.sh:1647 - Required criterion: “rerun first-only candidate failures so flakes are not findings.” The classifier treats any changed retry outcome as a flake. Thus an initial head failure followed by a timeout or runner error on retry is removed from regressed and can exit 0, although no passing retry established a flake. Keep such inconclusive retries refusing (or add a distinct refusing bucket); reserve flaky_transitions for a reproduced candidate that becomes passing.

🔧 Fix: Refuse inconclusive candidate retries
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• FM_TEST_RUN_ONLY=test_compare_commits_partitions_and_bounds_every_script bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_accounts_for_script_lost_after_discovery bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_rechecks_a_first_only_failure bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_refuses_an_inconclusive_retry bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_reports_timeout_to_failure_as_regressed bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_never_calls_deleted_or_skipped_tests_fixed bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_refuses_an_independent_enumeration_mismatch bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_headline_exposes_measured_coverage bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_preflight_errors_exit_two_without_tracebacks bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_signal_records_inflight_and_cleans_clones bash tests/fm-test-run.test.sh
• FM_TEST_RUN_ONLY=test_compare_commits_signal_records_inflight_confirmation bash tests/fm-test-run.test.sh
• Manual isolated --compare-commits fixtures retained CLI and JSON evidence for flake confirmation, timeout-to-failure regression, deletion/skipped erosion, enumeration refusal, and SIGTERM partial state/clone cleanup.

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.