fix(ci_status): report a failed CI query instead of faking "in_progress" - #146
Merged
Merged
Conversation
The Heart CI panel reported "18 repos need attention — CI in_progress" on machines with an older `gh`, and had done so for as long as that `gh` was installed. Nothing was in progress: the query itself was dead. `ci_status.sh` fetched runs with `gh run list --repo ... --branch main`. `--branch` only exists from gh 2.9, so on gh 2.4.0 the call exited non-zero, the trailing `|| echo '[]'` swallowed the error, and an empty run list rolled up to the `in_progress` default. The HEAD sha came from a separate `gh api` call that *did* work, so the sidecar looked freshly polled rather than broken. The cloud heart-health job runs a modern `gh`, so CI never saw it. Two independent defects, fixed separately: 1. Version fragility. Read runs from `gh api .../actions/runs?branch=main` instead of `gh run list`. The REST endpoint is stable across every `gh` that has `gh api` at all — the same call already used for the HEAD sha — whereas `gh run list`'s flags and `--json` field names have moved between releases. `normalize_runs` accepts both the REST payload and the legacy `gh run list` shape, so cached payloads keep working. REST reports the workflow display name in `name` (commit subject moves to `display_title`), which is exactly the key `required_workflows` is written against. 2. Silent degradation — the more dangerous half. A failed fetch was indistinguishable from genuinely pending CI, so a real red and a dead query rendered identically. A fetch failure now records `status="unavailable"` plus the error, surfaced as "CI UNAVAILABLE" in the log, "CI unavailable (query failed)" on the dashboard, and an explicit STALE reason in readiness. It is still never green, so no gate loosens — an unknown is now merely *stated* rather than disguised. Verified end-to-end against a stubbed `gh` in both modes: a working `gh` resolves the required workflow and rolls up to success; a 2.4.0-style failure reports UNAVAILABLE with the underlying error. Full suite: 469 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0194NVZM3fi79ErZ3zooCZWq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Heart CI panel has been reporting
LIBRARIES 7 repos, 7 need attention / WORKSPACES 11 repos, 11 need attention — CI in_progressfor 18/18 repos on machines with an oldergh, and had done so for as long as thatghwas installed. Nothing was in progress: the query itself was dead.Root cause
ci_status.shfetched runs withgh run list --repo … --branch main.--branchonly exists from gh 2.9, so on gh 2.4.0 the call exited non-zero, the trailing|| echo '[]'swallowed the error, and the empty run list rolled up torollup()'sin_progressdefault.head_shacame from a separategh apicall that did work, so the sidecar looked freshly polled rather than broken. The cloudheart-healthjob runs a moderngh, so CI never saw it.Confirmed against a real repo:
PyAutoLens.ci_status.jsonreportedstatus: in_progress, workflows: {}, while the actual check-runs on that same head sha were all green.Two independent defects, fixed separately
1. Version fragility. Runs are now read from
gh api …/actions/runs?branch=main&per_page=30&exclude_pull_requests=true. The REST endpoint is stable across everyghthat hasgh apiat all — the same call already used for the HEAD sha — whereasgh run list's flags and its--jsonfield names have moved between releases (workflowNameis newer thanname).normalize_runs()accepts both the REST payload and the legacygh run listlist, so cached payloads and existing callers keep working. Verified against the live REST response that a run's workflow display name is inname("Smoke Tests","Navigator Check") with the commit subject indisplay_title— i.e. exactly the keyrequired_workflowsis written against.2. Silent degradation — the more dangerous half. A failed fetch was indistinguishable from genuinely pending CI, so a real red and a dead query rendered identically. A fetch failure now records
status="unavailable"plus the underlying error, surfaced as:CI UNAVAILABLE <error>in the daemon logCI unavailable (query failed)on the dashboardreadiness(<lib>: CI status unavailable (…))It is still never green, so no gate loosens — an unknown is now merely stated rather than disguised. Unparseable stdin is treated the same way instead of silently becoming an empty run list.
Verification
test_ci_status.py,test_dashboard.py,test_readiness.py), covering REST parsing, legacy-shape compatibility, branch filtering, and that an errored fetch is neversuccessand neverin_progress.ghin both modes: a workingghresolves the required workflow and rolls up tosuccess; a 2.4.0-style failure reportsCI UNAVAILABLEwith the real error.Reviewer note: this may make the verdict look worse before better
This is a truthfulness fix, not a verdict-improving one. Today every repo rolls up to
conclusion="", which generates no library-CI red at all. Once real conclusions come back, any genuinely failing required workflow becomes a RED that was previously invisible. That is the point — a genuine red currently looks identical to this bug — but expect the honest score to move.Audited the sibling
ghcallers while here:release_run.pyandopen_prs.shuse only flags present in gh 2.4.0, consistent with those signals having worked while CI did not. Two follow-ups deliberately left out of this diff:agents/conductors/release/nightly.shuses--created(Actions-only, and it fails loudly rather than silently), andopen_prs.shcarries the same|| echo '[]'swallow pattern, where a failed query would read as "no open PRs".Generated by Claude Code