Repository navigation
Canary: report the question run's verdict even after a budget failure - #21
Merged
Merged
Conversation
The "Interpret the exit code" step had no if:, so when the proxy-traffic step before it failed (budget exceeded) GitHub skipped it and the run's own exit-code verdict never appeared. It now runs unless the job was cancelled or the run step itself was skipped; the job stays red either way. A smoke check requires every step that judges a run's exit code to carry !cancelled() or always(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In the canary's question half, "Interpret the exit code" had no
if:, so it carried GitHub's implicitsuccess(). When the "Proxy traffic of the question run" step before it failed (the run exceeded its proxy budget), GitHub skipped the interpret step. The log then showed only the budget error and never the run's own verdict (exit 6 partial, 5 content never obtained, and so on). The topic half's verdict steps already run after a failure (!cancelled()).Change
.github/workflows/canary.yml: "Interpret the exit code" now hasif: ${{ !cancelled() && steps.run.outcome != 'skipped' }}, the same guard the topic half uses. The job's result is unchanged: a failed proxy-traffic step still fails the job, and the interpret step still exits 1 on any nonzero scraper exit.smoke_test.py: a check that every step judging a run's.outputs.exit_codecarries!cancelled()oralways()(3 such steps today), and that the question run's interpret step is one of them.[Unreleased]/ Changed (CI only).Evidence
requirements.txt):python smoke_test.pyreportssmoke_test: OK(engine-only groups skipped). The workflow parses as YAML, and its triggers are still onlyscheduleandworkflow_dispatch.if:removed from "Interpret the exit code": "every exit-code verdict step runs after a failed proxy-traffic step (unguarded ['Interpret the exit code'])" fails, and so does "the question run's 'Interpret the exit code' step is one of them";!cancelled()removed from "Fail when the topic run did not parse rows": the same check fails, naming that step;--proxy-budget-mb 10: "every canary scrape passes --proxy-budget-mb 25" fails (the CI: run the canary with a 25 MB proxy budget #20 guard is still live).Not changed
The budget (25 MB, from #20), the scraper's 10 MB default, "Sanity-check the question run" (it runs only after exit 0, and a run over its budget never exits 0), and the release/version.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8