Repository navigation
fix: trigger red-flags on open-ended whole-file review prompts - #11
Merged
Merged
Conversation
Closes #9. "Anything off in this user service?" and "Can you take a look at compute_price?" fire no skill. The #10 wording made this worse: it rewrote the red-flags triggers into explicit-ask shapes ("asks for a scan", "asks to check against a checklist"), left the one open-ended trigger untouched, and added an exclusion. 09-layered-function went from 1/3 to 0/3 fired as a result. red-flags now claims open-ended review of a named or pasted file, class, or function, with an explicit guard so a specific bug or error the user already identified still routes elsewhere (06-neg-keyerror). design-review gets the matching exclusion so the two don't contend, the same shape as the #4 resolution. This invalidates the red-flags and design-review baselines. Verify with: claude plugin eval . --eval-dir evals/red-flags --ablation with-without --judge-model sonnet claude plugin eval . --eval-dir evals/design-review --ablation with-without --judge-model sonnet 08 and 09 should fire 3/3, 06-neg-keyerror should stay at 0 Skill calls, and design-review should not lose cases to red-flags.
Full runs of both suites at ad561ff, plus a traced re-run of 05-clean-py. red-flags: mean delta +0.20 -> +0.22. The #9 targets are fixed — 08-buried-passthrough went 0/3 fired and +0.18 to 3/3 and +0.79, the largest uplift in the suite; 09-layered-function 0/3 to 3/3 at +0.36. 06-neg-keyerror still fires nothing, so the guard clause holds. Every should-fire case now fires. design-review: mean delta flat at +0.08. 04-clean-go's earlier 0.50/0.50 was noise and recovered to 1.00/0.50 with no change behind it. 05-clean-py was not noise. Traces show red-flags firing instead of design-review in 1 of 3 runs on "Review the design of slugify.py", padding the clean file for a 0.50 where the design-review runs score 1.00. The #9 exclusion said "a single file or function", which is too broad: it swallows an explicit design review of one file. It now keys on a prompt that names no review goal. This narrowing is unvalidated; one design-review run (~$8, 9 min) should confirm the steal is gone. Also documented: this suite's skill-fired grader carries input_match: design-review, while the red-flags suite's grader counts any skill call. The two columns are not comparable, and 08/09 would be unambiguous with input_match: red-flags added.
05-clean-py at fd78f9c: 6/6 runs fire design-review, none fire red-flags, scoring 0.92/0.67 (delta +0.25, up from -0.17). The steal the previous traces caught is gone. One of the six still fails clean-verdict by padding with design-review correctly fired, which is the ordinary failure this case exists to grade.
codybrom
added a commit
that referenced
this pull request
Oct 8, 2026
Plugin: the skill trigger and design-it-twice fixes from #10 and #11. Site: the MCP server at clairvoyance.fyi/mcp (listed in the official MCP Registry), /install and agent setup, and agent discovery. See CHANGELOG.md. Also links the server's registry entry from /mcp and the README, and describes the server as "inspired by" the book.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9.
Problem
Generic whole-file review prompts fired no skill. The wording in #10 made it worse: it rewrote the
red-flagstriggers into explicit-ask shapes ("asks for a scan", "asks to check against a checklist"), left the one open-ended trigger untouched, and added an exclusion.09-layered-functionwent from 1/3 to 0/3 fired as a result.Changes
red-flagsclaims open-ended review of a named or pasted file, class, or function, with an explicit guard so a specific bug or error the user already identified still routes elsewhere.design-reviewgets the matching exclusion so the two do not contend, the same shape as the No skill triggers on 'review this diff before I merge' #4 resolution. It keys on a prompt that names no review goal — an earlier draft said "a single file or function", which was too broad and letred-flagssteal explicit design reviews (see below).Results
red-flags, full run atad561ff, mean Δ +0.20 → +0.22:08-buried-passthrough09-layered-function06-neg-keyerrorEvery should-fire case now fires 3/3, and the negative case still fires nothing. 08 is the largest uplift in the suite.
design-review, full run atad561ff, mean Δ flat at +0.08. Two findings came out of it:04-clean-go's apparent regression in the previous baseline was noise — it recovered 0.50/0.50 → 1.00/0.50 with nothing changed behind it.05-clean-pywas not noise. Traces showedred-flagsfiring instead ofdesign-reviewon "Review the design of slugify.py" in 1 of 3 runs, padding the clean file for a 0.50. That is what prompted narrowing the exclusion. A traced 6-run re-run atfd78f9cconfirms the fix: 6/6design-review, nored-flags, 0.92 / 0.67, Δ +0.25.Notes
The
skill-firedgraders differ between the suites:design-review's carriesinput_match: design-review, whilered-flags' counts any skill call. So "08 and 09 fire 3/3" means a skill fired. The scores against red-flags-specific graders (service-passthrough,decorator-forwarding) make it near-certain it is the right one, but addinginput_match: red-flagsto those two cases would make it unambiguous. Not done here.Verify