Automate CodeRabbit rate-limit fallback to exact-head Qodo review

[safal207]

Goal

Add a safe reviewer fallback for pull requests when CodeRabbit reports a native review-rate limit.

Qodo is the primary exact-head reviewer. CodeRabbit remains supplemental. A CodeRabbit quota failure must not be treated as approval and must not stall the review graph when trusted Qodo review is available.

Trigger

React only to a trusted native CodeRabbit bot comment that explicitly reports review unavailability or rate limiting, such as Review limit reached.

Required trust checks

Before requesting Qodo review, the automation must verify:

1. repository and pull request are the expected target;
2. pull request is still open;
3. event actor/comment author is the canonical CodeRabbit bot identity;
4. rate-limit marker is present in the trusted bot payload;
5. the recorded PR head is a full 40-character SHA and still equals the current head;
6. no successful or in-progress Qodo request already exists for that exact head;
7. no newer head superseded the request;
8. the workflow cannot be triggered recursively by its own comment.

Fallback action

Post exactly one native Qodo /review request bound to the current PR head, plus a structured review rubric:

• reconstruct intended invariants first;
• treat green CI as evidence, not proof;
• search for fail-open behavior, stale-SHA acceptance, artifact substitution, symlink/path ambiguity, duplicate-key ambiguity, mutable-reference downgrade, cross-run evidence mixing, nondeterminism, and unsafe error handling;
• require a concrete counterexample for each suspected issue;
• report only actionable correctness, security, determinism, or trust-contract findings;
• for each finding include severity, affected guarantee, exact boundary, failure path, minimal regression test, and minimal remediation;
• explicitly bind a no-findings result to the exact reviewed SHA.

This is a Claude-like skeptical review method, not a claim that Qodo runs a specific model.

Evidence output

Publish or update one canonical machine-readable status containing:

• schema version;
• repository and PR number;
• exact head SHA;
• CodeRabbit status and trusted comment ID;
• Qodo request status and request comment ID;
• request timestamp;
• stale/superseded flag;
• final Qodo review identity, SHA, and outcome when available.

Fail-closed behavior

• spoofed/non-bot rate-limit comments: reject;
• missing or short SHA: reject;
• stale head: reject;
• duplicate request for the same SHA: no-op;
• Qodo unavailable: record PROVIDER_EVIDENCE_UNAVAILABLE, never approval;
• workflow error: publish diagnostic evidence, never approval;
• no merge or auto-merge authority.

Acceptance criteria

• deterministic tests for trusted rate-limit, spoofed actor, stale head, duplicate delivery, concurrent delivery, superseded request, Qodo unavailable, and successful exact-head request;
• least-privilege workflow permissions;
• immutable action pins;
• no pull_request_target execution of untrusted PR code;
• exact-run/exact-attempt evidence;
• no comment loops or duplicate Qodo reviews.

Related implementation context: PR #167.

[safal207]

Implementation merged; live validation pending

PR #174 was squash-merged into main.

• reviewed PR head: 84d71958be9b8b224e20640eb134ae775a47666b;
• merge commit: c5bd3689453baf4f022cdc9ffaa155a63724713f;
• CI: 29366019603 — success;
• Python Package Validation: 29366020188 — success;
• Security Baseline: 29366020336 — success;
• 465 tests per Python 3.10/3.11/3.12 lane, 0 failures/errors/skips, coverage 76.7%;
• Qodo exact-head review: no new actionable security findings;
• CodeRabbit exact-delta review: no actionable comments;
• unresolved review threads: 0;
• explicit bootstrap disposition: PR comment 4973826212;
• Trust Root Gate rejection contained only the five expected protected-bootstrap findings.

The protected workflow is now present on main at merge commit c5bd3689453baf4f022cdc9ffaa155a63724713f with workflow blob d5fba1f0e7daf13a3fe3e9c44af2d08890f3c888.

This issue remains open intentionally. Final acceptance still requires a real post-merge live event where canonical CodeRabbit emits an actual rate-limit/unavailability comment and the workflow proves:

1. exactly one Qodo request is created for the current exact head;
2. request/result artifacts bind to the correct run and attempt;
3. no successful merge-authority status is emitted;
4. replay, edited-comment, and stale-head behavior remain fail-closed.

[safal207]

Live event diagnosed; remediation PR opened

The real CodeRabbit rate-limit event observed on PR #176 did not create either canonical marker:

• no cml-reviewer-fallback-status:v2 comment;
• no authenticated cml-qodo-fallback-request:v2 comment.

The merged implementation had two event-surface gaps:

1. .github/workflows/reviewer-fallback.yml listened only to issue_comment, while native providers can emit review unavailability through pull_request_review or pull_request_review_comment;
2. the artifact authenticator required the workflow run event to equal issue_comment, so adding triggers alone would still reject legitimate review-surface artifacts during replay/result validation.

Draft remediation: #177

The patch adds one protected event adapter for all three native surfaces while preserving the unchanged strict core state machine. It also prevents inline Qodo comments from completing a review lifecycle and adds regressions for trusted, spoofed, inline, unsupported, and exact-head paths.

Issue #168 remains open. Even after #177 merges, final acceptance still requires the next real post-merge rate-limit event to prove exactly one Qodo request, exact-run/exact-attempt evidence, no success merge status, and fail-closed replay behavior.

[safal207]

Native event-surface remediation merged

PR #177 has been squash-merged into main.

• validated head: 86a603218ec9ee3950e54118596b051a5143fefd;
• immutable merge commit: cc77c17f86b43715de0d3c8dab164712ef3c730f;
• CI / package / security: success;
• 505 tests per Python 3.10 / 3.11 / 3.12 lane;
• coverage: 77.72%;
• Qodo and CodeRabbit exact-head reviews: no actionable findings;
• protected trust-root bootstrap: explicitly dispositioned with the strict core unchanged.

The merged workflow and adapter now cover CodeRabbit rate-limit output from:

• issue_comment;
• pull_request_review;
• pull_request_review_comment.

Complete Qodo results are accepted only from top-level created comments or submitted reviews. Qodo inline comments do not start the normal write-capable job and cannot complete the lifecycle if presented directly.

Why this issue remains open

The exact historical webhook type for the Review limit reached event on PR #176 could not be conclusively recovered from the metadata available to this investigation. We proved and closed real event-surface blind spots, but that is not equivalent to proving the full production path with a fresh live event.

Final acceptance still requires the next genuine post-merge CodeRabbit rate-limit event to demonstrate, on one unchanged exact head:

1. exactly one authenticated Qodo request;
2. one canonical lifecycle status comment;
3. exact-run / exact-attempt artifact evidence;
4. no successful merge-status publication;
5. duplicate/replayed delivery becoming a deterministic no-op or fail-closed result;
6. any stale or superseded head being rejected.

Until that happens, provider unavailability remains evidence of unavailability—not approval—and #168 stays open.

[safal207]

Second genuine post-merge rate-limit event: fallback still not observed

PR #164 exact head 5fe200c2c1e27c3867f581f7f218846ef7a4ad63 produced a genuine CodeRabbit Review limit reached message after the PR was marked ready.

The provider updated its existing top-level issue comment (4827658127), so the GitHub surface was an edited issue_comment. The merged workflow declares issue_comment: [created, edited], and the strict core explicitly accepts action in {created, edited} for CodeRabbit while rejecting edited Qodo results only.

Observed result:

• no cml-qodo-fallback-request:v2 marker;
• no cml-reviewer-fallback-status:v2 lifecycle comment;
• no exact-head fallback evidence visible on the PR;
• no successful fallback status was published.

Therefore the live acceptance criteria are still unmet. The remaining failure is upstream of, or at, Actions workflow delivery/execution rather than the core CodeRabbit routing branch. Do not close #168 or treat this provider failure as approval.

The event is preserved on PR #164 for further workflow-delivery diagnosis.

[safal207]

Fresh post-merge live event on PR #178 — fallback not yet accepted

A genuine canonical CodeRabbit rate-limit comment occurred on PR #178 after the event-surface remediation in #177 had reached main.

Live event

• PR: docs: verify CML Trust Root Gate protection #178;
• exact head: fe96b454a95864d9d294712f7c2d5ccd0123f14a;
• CodeRabbit comment ID: 4996684346;
• author: coderabbitai[bot], numeric ID 136622811;
• marker: Review limit reached / couldn't start this review.

Important ordering observation

Qodo had already started its ordinary automatic review before the CodeRabbit limit comment:

• Qodo working comment: 4996684216;
• CodeRabbit limit comment: 4996684346.

Therefore the later Qodo summary/final comments cannot by themselves prove that the protected fallback created the review request.

Evidence observed

• Qodo summary comment: 4996688124;
• Qodo final comment: 4996692896 — zero bugs, zero rule violations, zero requirement gaps;
• exact-head CI 29535536574: success;
• package validation 29535536543: success;
• security baseline 29535536592: success;
• Trust Root Gate run 29535536540, attempt 1: success;
• commit statuses exposed only CodeRabbit and CML Trust Root Gate.

Missing acceptance evidence

The PR conversation did not expose the two protected Actions-authored lifecycle records required by the v2 contract:

• no observable authenticated cml-qodo-fallback-request:v2 request comment;
• no observable canonical cml-reviewer-fallback-status:v2 status comment.

Because Qodo was already running independently, its clean result is not sufficient fallback evidence. The live event therefore does not satisfy #168 acceptance and must not be interpreted as approval.

Disposition

Keep #168 open and fail closed. The next investigation should identify whether the issue_comment workflow was not dispatched, was skipped by the workflow-level identity predicate, or failed before publishing its protected request/status lifecycle. Preserve the successful property that no CML Reviewer Fallback success merge status was emitted.

[safal207]

Missed-event recovery PR opened

Draft PR #179 implements a fail-closed recovery path for the production gap observed on PR #178.

• current exact head: 786034d40bb61b436103923e2b377d6882d1f91d;
• CI run 29537160971: success;
• Python Package Validation run 29537160956: success;
• Security Baseline run 29537160984: in progress at the time of this update;
• Trust Root Gate run 29537161042: expected protected-bootstrap rejection;
• trust-root artifact: cml-trust-root-pr179-786034d40bb61b436103923e2b377d6882d1f91d-29537161042-1;
• artifact digest: sha256:6ae8719bbf0b304d2e0ccf96188a2fcbb862478218e3d2d303321bcc5c6ca142.

The recovery design uses a two-stage model:

1. an hourly discovery job with no issue/status write authority finds only canonical CodeRabbit rate-limit comments bound to the current full SHA;
2. it dispatches the same protected workflow with PR + comment ID, so the write-capable recovery run shares the same per-PR concurrency lock as native webhook delivery.

The strict core remains the final exactly-once decision point. workflow_dispatch request artifacts are authenticated; scheduled discovery artifacts are explicitly excluded from request evidence. Successful recovery still emits no success status and retains merge_authority: false.

Issue #168 remains open. After this protected bootstrap is reviewed and merged, final acceptance still requires a fresh live CodeRabbit rate-limit event on an open unchanged exact head.

[safal207]

Protected missed-event recovery merged

PR #179 has been squash-merged into main.

• validated exact head: 786034d40bb61b436103923e2b377d6882d1f91d;
• immutable merge commit: 2b6d69540096b42313979dc8fcaed526caf14bd7;
• CI run 29537160971: success;
• Python Package Validation run 29537160956: success;
• Security Baseline run 29537160984: success;
• unresolved review threads: 0;
• Qodo exact-head proxy conclusion: no actionable findings;
• CodeRabbit exact-head analysis: all 23 protected blob pins matched, with no success-status or merge-authority path;
• expected Trust Root Gate bootstrap run: 29537161042, attempt 1;
• exact-attempt artifact: cml-trust-root-pr179-786034d40bb61b436103923e2b377d6882d1f91d-29537161042-1;
• artifact digest: sha256:6ae8719bbf0b304d2e0ccf96188a2fcbb862478218e3d2d303321bcc5c6ca142.

The merged design adds hourly read-oriented discovery and a separately dispatched write-capable reconciliation run. Native webhook and recovery delivery serialize on the same per-PR concurrency key; the protected strict core remains the final exact-head and exactly-once decision point. Scheduled discovery artifacts cannot authenticate request lifecycle evidence, successful recovery never emits a success merge status, and merge_authority remains false.

Why #168 remains open

The rate-limit event observed on PR #179 happened before this recovery tree reached main, so it cannot validate the merged recovery implementation. Final acceptance still requires the next genuine post-merge CodeRabbit rate-limit event on an open unchanged PR head to prove:

1. one authenticated exact-head Qodo request;
2. one canonical lifecycle status;
3. exact-run/exact-attempt request and result artifacts;
4. no successful merge-authority status;
5. deterministic duplicate/replay no-op;
6. stale or superseded head rejection.