fix: "could not tell at all" exits non-zero on every lane#41
Merged
Conversation
… codes Fixes the six edges from the 2026-07-23 error-vs-fail audit, all in the inconclusive-maps-to-exit-0 direction; every change tightens toward red and none weakens fail-closed semantics: 1. CRITICAL `hotato run --scenarios/--audio`: process_exit_code now maps ANY envelope whose every event is not scorable (single or suite) to exit 2, and run_suite refuses a user scenarios dir with zero scenario files -- an emptied battery / wrong --audio path / un-fetched CI artifacts read red, mirroring prove's "a proof of nothing is refused". The MCP run tool inherits the same mapping. A battery with at least one scored event keeps its scored verdict. 2. `hotato pull --score`: the aggregate follows process_exit_code per recording -- a pulled set in which not one recording produced a scorable event exits 2 (skips and not-scorables beside scored results stay per-file reports). 3. rubric lane: a response that never parses into a categorical verdict even after the repair retry (empty/garbage body) raises JudgeError -> ERROR, never an "inconclusive" abstention vote, so it blocks under gating. 4. Gated judge ERROR with no rubric FAIL exits 2 (refuse), distinct from a scored FAIL's exit 1, in rubric run --gate and test run --gate-judge. 5. `hotato drive`: a fresh call with no scorable moment exits 2 (unusable fresh evidence, still red), reserving exit 1 for a scored invariant FAIL. 6. `hotato pull` / `hotato sweep`: every listed recording failing to fetch exits 2 -- an outage, not a completed pull/sweep. Regression tests per finding in tests/test_fail_closed_edges.py (each red on the prior behavior), plus the mixed-batch pins: scored results keep their scored verdict semantics. Exit-code docs (cli epilogs/describe, RUBRIC, BAD-CALL-TO-CI, CONNECT) updated; llms-full.txt rebuilt. Claude-Session: https://claude.ai/code/session_015njkxtoAKioqTPT2k14xFr
hotato turn-taking eval8 of 8 scenarios pass. 0 fail. No regression.
RegressionsNone. Reproducible timing measured locally from call audio. Swap the bundled self-test step for your own captured recordings to gate on your agent. github.com/attenlabs/hotato |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the six audited edges where an inconclusive result could exit 0, extending prove's inconclusive-never-green stance to every lane:
hotato run --scenarios/--audio) now exits 2 (was 0); MCP run tool inherits the mappingpull --scoreexits 2 when not one pulled recording produced a scorable event--gate/--gate-judgedrivemaps a not-scorable outcome to exit 2, reserving exit 1 for scored invariant failurespull/sweepexit 2 when every listed call failed to fetchScored semantics are unchanged: a battery or pulled set with at least one scored event keeps its scored verdict (pinned by 5 keep-scored-semantics tests); a well-formed INCONCLUSIVE verdict stays advisory. 17 new tests in tests/test_fail_closed_edges.py (12 reproduce the old behavior red-on-main); docs (RUBRIC, BAD-CALL-TO-CI, CONNECT) and llms-full regenerated.
Independently verified by a fresh reviewer: every finding reproduced on main and gated on this branch (including MCP parity), 13 over-tightening guard cases byte-identical across main/branch, full suite green, zero fail-closed lines removed.
Audit: hotato-growth/research/ERROR-VS-FAIL-AUDIT-2026-07-23.md
https://claude.ai/code/session_015njkxtoAKioqTPT2k14xFr