What happened
The review agent on PR #60 identified two valid documentation gaps in SPEC.md (the default settings table and valid keys list were missing hide_stale_sessions). However, since SPEC.md was not in the PR diff, both findings were omitted from the PR review per the post-review script logic: '2 findings omitted because the files were not in the PR diff.' The findings were still posted as an issue comment, but the follow-up issue creation mechanism is disabled per #1137. This means the agent spent tokens identifying real issues that have no path to resolution.
What could go better
Valid findings in non-diff files should have an actionable path. Currently they're posted in a comment that's easily overlooked and can't become tracked follow-up issues. This is a known gap (issue #1137 exists), but it means the review agent is doing useful work that gets dropped on the floor. Confidence is moderate — the findings are real but low-severity (doc updates), so the practical impact of losing them is limited. However, for higher-severity findings in non-diff files, this pattern could miss real problems.
Proposed change
This is primarily a priority signal for #1137 (follow-up issue creation). Until that's resolved, consider a lighter-weight alternative: when findings are omitted from the PR review, include them in the review comment body under a clearly labeled 'Out-of-scope findings' section (rather than silently omitting them), so the PR author and reviewers can see them even if they're not inline on the diff. This would be a change to the post-review script's comment formatting logic.
Validation criteria
After the change, review comments on PRs where non-diff findings exist should include an 'Out-of-scope findings' section listing those findings. Verify on the next 3 PRs where the review agent identifies issues in files outside the diff.
Generated by retro agent from agentshed/seshi#60
What happened
The review agent on PR #60 identified two valid documentation gaps in
SPEC.md(the default settings table and valid keys list were missinghide_stale_sessions). However, sinceSPEC.mdwas not in the PR diff, both findings were omitted from the PR review per the post-review script logic: '2 findings omitted because the files were not in the PR diff.' The findings were still posted as an issue comment, but the follow-up issue creation mechanism is disabled per #1137. This means the agent spent tokens identifying real issues that have no path to resolution.What could go better
Valid findings in non-diff files should have an actionable path. Currently they're posted in a comment that's easily overlooked and can't become tracked follow-up issues. This is a known gap (issue #1137 exists), but it means the review agent is doing useful work that gets dropped on the floor. Confidence is moderate — the findings are real but low-severity (doc updates), so the practical impact of losing them is limited. However, for higher-severity findings in non-diff files, this pattern could miss real problems.
Proposed change
This is primarily a priority signal for #1137 (follow-up issue creation). Until that's resolved, consider a lighter-weight alternative: when findings are omitted from the PR review, include them in the review comment body under a clearly labeled 'Out-of-scope findings' section (rather than silently omitting them), so the PR author and reviewers can see them even if they're not inline on the diff. This would be a change to the post-review script's comment formatting logic.
Validation criteria
After the change, review comments on PRs where non-diff findings exist should include an 'Out-of-scope findings' section listing those findings. Verify on the next 3 PRs where the review agent identifies issues in files outside the diff.
Generated by retro agent from agentshed/seshi#60