Skip to content

fix(crosscheck): use provider-valid verdict repair effort - #335

Merged
ruby-dlee merged 1 commit into
mainfrom
fm/crosscheck-verdict-effort-r9
Aug 25, 2026
Merged

fix(crosscheck): use provider-valid verdict repair effort#335
ruby-dlee merged 1 commit into
mainfrom
fm/crosscheck-verdict-effort-r9

Conversation

@ruby-dlee

Copy link
Copy Markdown
Owner

Intent

Fix the released PR 333 verdict-repair incompatibility that blocks PR 327's exact-head Azure Crosscheck. The exact live failure was at PR 327 head 85f89f8: the primary model review completed at xhigh, then the one bounded verdict repair sent reasoning_effort=minimal and direct Fireworks rejected that literal because it accepts low, medium, high, xhigh, max, none, adaptive, integer, or boolean. Reproduce the failure through deterministic adapter/guest tests, identify one owner of repair effort, and make the smallest provider-valid correction. Do not lower the primary xhigh review, weaken schema or evidence gates, add retries, change PR 327 source, touch cloud-result-return files, or invoke another paid review of PR 327. Preserve exactly one bounded fresh repair and terminal telemetry. Add a regression that fails on literal minimal and executes the production command-building/parser path. Validate focused Crosscheck and Azure model-compartment tests, lint, compilation, and docs, then open a green PR and obtain exact-head Crosscheck for that new PR. PR 327's owner will restart its already-authorized review only after this correction lands and deploys.

What Changed

  • Use the provider-supported low reasoning effort for the single bounded Pi verdict-repair attempt while preserving the primary review effort.
  • Extend the Azure Crosscheck adapter regression to reject unsupported effort values and verify the production repair command and prompt use low instead of minimal.
  • Update Crosscheck documentation to describe the low-reasoning repair attempt.

Risk Assessment

✅ Low: Captain, the change is narrowly scoped, preserves the primary xhigh review and single bounded repair, and exercises the production command-building/parser path with provider-valid low effort.

Testing

The previously successful baseline plus the focused Azure Crosscheck suite and a deterministic production-path replay all passed; reviewer-visible evidence confirms xhigh primary effort, exactly one low-effort repair, successful verdict parsing, and preserved terminal telemetry.

Evidence: Deterministic end-to-end verdict-repair transcript
{
"user_flow": "xhigh primary protocol miss -> one fresh low repair -> accepted verdict",
"exit_code": 0,
"attempts": 2,
"thinking_efforts": ["xhigh", "low"],
"verdict": {"summary": "clear"},
"telemetry_turns": 2,
"telemetry_tokens": {
"input": 20,
"output": 4,
"cache_read": 8,
"cache_write": 0,
"source": "pi-turn-end-message-usage"
}
}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • if [ "${FM_AZURE_VALIDATION_CELL:-0}" = 1 ]; then exec "$FM_AZURE_VALIDATION_SHARD_BRIDGE" behavior --count "${FM_AZURE_VALIDATION_SHARD_COUNT:-8}"; else exec bin/fm-no-mistakes-test-command.sh; fi
  • Baseline configured test command (reported as already successful by the outer executor).
  • tests/fm-crosscheck-azure.test.sh
  • Executed the regression’s deterministic fake-provider harness through bin/fm-crosscheck-pi-reviewer.py and bin/fm-crosscheck-pi-verdict-extension.ts using terminal-error-then-valid, then emitted the captured command efforts, verdict, and telemetry.
  • git status --short to confirm testing left no worktree artifacts.
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@ruby-dlee
ruby-dlee force-pushed the fm/crosscheck-verdict-effort-r9 branch from 6cf9cc7 to 1136611 Compare August 25, 2026 19:34
@ruby-dlee
ruby-dlee force-pushed the fm/crosscheck-verdict-effort-r9 branch from 1136611 to c0973c4 Compare August 25, 2026 21:10
@ruby-dlee
ruby-dlee merged commit ba0b898 into main Aug 25, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant