Skip to content

fix: prevent AI PR Review agents from failing checks on large PRs#2258

Merged
imran-siddique merged 1 commit into
microsoft:mainfrom
imran-siddique:fix/ai-pr-review-large-diffs
May 14, 2026
Merged

fix: prevent AI PR Review agents from failing checks on large PRs#2258
imran-siddique merged 1 commit into
microsoft:mainfrom
imran-siddique:fix/ai-pr-review-large-diffs

Conversation

@imran-siddique
Copy link
Copy Markdown
Member

Description

Fixes AI PR Review workflow checks (Code Review, Security Scan, Breaking Changes, Docs Sync, Test Coverage) showing as red X failures on large PRs like #2236.

Root Cause

  1. Diff truncation limit (24k chars) exceeded model context windows, causing HTTP 413 from GitHub Models API
  2. Fallback model (gpt-4o-mini) has only 8k token limit, too small for any PR review
  3. Catch block set \process.exitCode = 1, making informational checks appear as CI failures

Changes

  • Reduce diff truncation from 24k to 12k chars to stay within model context windows
  • 413 retry with truncation: on payload-too-large, halve the user prompt before retrying
  • Graceful failure: replace \process.exitCode = 1\ with ::warning::\ annotation so AI agents don't produce red X marks in the checks list
  • Better fallback: use gpt-4o as fallback instead of gpt-4o-mini

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Package(s) Affected

  • docs / root

Checklist

  • My code follows the project style guidelines (ruff check)
  • All new and existing tests pass (pytest)
  • I have signed the Microsoft CLA

AI Assistance

  • I can explain every meaningful change in this PR
  • I have run tests and verification appropriate for this change
  • No part of this PR was autonomously submitted by an AI agent without my review

GitHub Copilot assisted with drafting commit messages.

Three changes:
- Reduce diff truncation from 24k to 12k chars to stay within model
  context windows (the old limit caused 413 errors on large PRs)
- On 413 retry, halve the user prompt before retrying with fallback
  model so the retry has a chance of succeeding
- Replace process.exitCode=1 with a warning annotation so
  informational AI agents don't produce red X check marks
- Use gpt-4o as fallback instead of gpt-4o-mini (8k token limit
  was too small for PR review context)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@imran-siddique imran-siddique merged commit c95597e into microsoft:main May 14, 2026
30 of 31 checks passed
@imran-siddique imran-siddique deleted the fix/ai-pr-review-large-diffs branch May 14, 2026 14:43
@github-actions
Copy link
Copy Markdown

🤖 AI Agent: test-generator — `.github/actions/ai-agent-runner/action.yml`

.github/actions/ai-agent-runner/action.yml

  • test_diff_truncation -- Validate that diffs exceeding 12k characters are truncated correctly.
  • test_413_retry_logic -- Ensure the retry mechanism halves the user prompt on HTTP 413 errors.
  • test_graceful_failure -- Confirm that errors log warnings instead of failing the workflow.

.github/workflows/ai-pr-review.yml

  • test_fallback_model_switch -- Verify that the fallback model switches to gpt-4o instead of gpt-4o-mini.

@github-actions github-actions Bot added the size/S Small PR (< 50 lines) label May 14, 2026
@github-actions
Copy link
Copy Markdown

🤖 AI Agent: security-scanner — View details

No security issues found.

@github-actions
Copy link
Copy Markdown

🤖 AI Agent: breaking-change-detector — API Compatibility

API Compatibility

No breaking changes detected.

@github-actions
Copy link
Copy Markdown

🤖 AI Agent: code-reviewer — Action Items:

TL;DR: 0 blockers, 1 warning. Fix improves robustness of AI PR review checks but requires further testing for edge cases.

# Sev Issue Where
1 Warn Edge cases for 413 retry truncation logic .github/actions/ai-agent-runner

Action Items:

  • None.

Warnings:

# Issue Where Follow-up
1 Edge cases for 413 retry truncation logic .github/actions/ai-agent-runner Fine as follow-up PRs.

@github-actions
Copy link
Copy Markdown

🤖 AI Agent: docs-sync-checker — Docs Sync

Docs Sync

Documentation is in sync.

@github-actions
Copy link
Copy Markdown

PR Review Summary

Check Status Details
🔍 Code Review ⚠️ Warning See details
🛡️ Security Scan ✅ Passed No issues found
🔄 Breaking Changes ✅ Passed No issues found
📝 Docs Sync ✅ Passed No issues found
🧪 Test Coverage ✅ Completed Analysis complete

Verdict: ⚠️ Ready for human review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scripts/ci/cd size/S Small PR (< 50 lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant