Skip to content

fix(ci_status): report a failed CI query instead of faking "in_progress" - #146

Merged
Jammy2211 merged 1 commit into
mainfrom
claude/release-blocking-issues-opn43d
Aug 16, 2026
Merged

fix(ci_status): report a failed CI query instead of faking "in_progress"#146
Jammy2211 merged 1 commit into
mainfrom
claude/release-blocking-issues-opn43d

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

The Heart CI panel has been reporting LIBRARIES 7 repos, 7 need attention / WORKSPACES 11 repos, 11 need attention — CI in_progress for 18/18 repos on machines with an older gh, and had done so for as long as that gh was installed. Nothing was in progress: the query itself was dead.

Root cause

ci_status.sh fetched runs with gh run list --repo … --branch main. --branch only exists from gh 2.9, so on gh 2.4.0 the call exited non-zero, the trailing || echo '[]' swallowed the error, and the empty run list rolled up to rollup()'s in_progress default. head_sha came from a separate gh api call that did work, so the sidecar looked freshly polled rather than broken. The cloud heart-health job runs a modern gh, so CI never saw it.

Confirmed against a real repo: PyAutoLens.ci_status.json reported status: in_progress, workflows: {}, while the actual check-runs on that same head sha were all green.

Two independent defects, fixed separately

1. Version fragility. Runs are now read from gh api …/actions/runs?branch=main&per_page=30&exclude_pull_requests=true. The REST endpoint is stable across every gh that has gh api at all — the same call already used for the HEAD sha — whereas gh run list's flags and its --json field names have moved between releases (workflowName is newer than name).

normalize_runs() accepts both the REST payload and the legacy gh run list list, so cached payloads and existing callers keep working. Verified against the live REST response that a run's workflow display name is in name ("Smoke Tests", "Navigator Check") with the commit subject in display_title — i.e. exactly the key required_workflows is written against.

2. Silent degradation — the more dangerous half. A failed fetch was indistinguishable from genuinely pending CI, so a real red and a dead query rendered identically. A fetch failure now records status="unavailable" plus the underlying error, surfaced as:

  • CI UNAVAILABLE <error> in the daemon log
  • CI unavailable (query failed) on the dashboard
  • an explicit STALE reason in readiness (<lib>: CI status unavailable (…))

It is still never green, so no gate loosens — an unknown is now merely stated rather than disguised. Unparseable stdin is treated the same way instead of silently becoming an empty run list.

Verification

  • Full suite: 469 passed (24 new tests across test_ci_status.py, test_dashboard.py, test_readiness.py), covering REST parsing, legacy-shape compatibility, branch filtering, and that an errored fetch is never success and never in_progress.
  • End-to-end against a stubbed gh in both modes: a working gh resolves the required workflow and rolls up to success; a 2.4.0-style failure reports CI UNAVAILABLE with the real error.

Reviewer note: this may make the verdict look worse before better

This is a truthfulness fix, not a verdict-improving one. Today every repo rolls up to conclusion="", which generates no library-CI red at all. Once real conclusions come back, any genuinely failing required workflow becomes a RED that was previously invisible. That is the point — a genuine red currently looks identical to this bug — but expect the honest score to move.

Audited the sibling gh callers while here: release_run.py and open_prs.sh use only flags present in gh 2.4.0, consistent with those signals having worked while CI did not. Two follow-ups deliberately left out of this diff: agents/conductors/release/nightly.sh uses --created (Actions-only, and it fails loudly rather than silently), and open_prs.sh carries the same || echo '[]' swallow pattern, where a failed query would read as "no open PRs".


Generated by Claude Code

The Heart CI panel reported "18 repos need attention — CI in_progress" on
machines with an older `gh`, and had done so for as long as that `gh` was
installed. Nothing was in progress: the query itself was dead.

`ci_status.sh` fetched runs with `gh run list --repo ... --branch main`.
`--branch` only exists from gh 2.9, so on gh 2.4.0 the call exited non-zero,
the trailing `|| echo '[]'` swallowed the error, and an empty run list rolled
up to the `in_progress` default. The HEAD sha came from a separate `gh api`
call that *did* work, so the sidecar looked freshly polled rather than broken.
The cloud heart-health job runs a modern `gh`, so CI never saw it.

Two independent defects, fixed separately:

1. Version fragility. Read runs from `gh api .../actions/runs?branch=main`
   instead of `gh run list`. The REST endpoint is stable across every `gh`
   that has `gh api` at all — the same call already used for the HEAD sha —
   whereas `gh run list`'s flags and `--json` field names have moved between
   releases. `normalize_runs` accepts both the REST payload and the legacy
   `gh run list` shape, so cached payloads keep working. REST reports the
   workflow display name in `name` (commit subject moves to `display_title`),
   which is exactly the key `required_workflows` is written against.

2. Silent degradation — the more dangerous half. A failed fetch was
   indistinguishable from genuinely pending CI, so a real red and a dead
   query rendered identically. A fetch failure now records
   `status="unavailable"` plus the error, surfaced as "CI UNAVAILABLE" in the
   log, "CI unavailable (query failed)" on the dashboard, and an explicit
   STALE reason in readiness. It is still never green, so no gate loosens —
   an unknown is now merely *stated* rather than disguised.

Verified end-to-end against a stubbed `gh` in both modes: a working `gh`
resolves the required workflow and rolls up to success; a 2.4.0-style failure
reports UNAVAILABLE with the underlying error. Full suite: 469 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194NVZM3fi79ErZ3zooCZWq
@Jammy2211
Jammy2211 merged commit 764a3f8 into main Aug 16, 2026
2 checks passed
@Jammy2211
Jammy2211 deleted the claude/release-blocking-issues-opn43d branch August 25, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants