Repository navigation
fix: sharpen skill triggers and scope clean-room dispatch - #10
Merged
Merged
Conversation
- code-evolution owns plain diff/PR review; red-flags and design-review defer to it (#4) - design-it-twice description covers "how should I structure X" and RFC/design-doc/ADR prompts (#5) - design-it-twice: isolation mode only for a user- or codebase-supplied design; add a convergence check that falls back to the pre-mortem (#6) - evals: raise 01-rate-limiter timeout to 600s so clean-room runs finish
codybrom
added a commit
that referenced
this pull request
Sep 19, 2026
The trigger fixes in #10 changed three suites' numbers and no README was updated. These tables come from the full 3-run runs already in evals/*/results/, whose skill files are byte-identical to b55f118. - red-flags: mean delta +0.13 -> +0.20. 03-diff-review is fixed (0/3 -> 3/3 fired, +0.12 -> +0.60). 08 and 09 are now described as the #9 gap; 09 regressed from 1/3 to 0/3 fired. - design-it-twice: gains the Baseline and When-to-re-run sections it never had. Mean delta +0.18 -> +0.36. 01-rate-limiter went from -0.11/-0.44 to +0.05 and 04-retry-rfc from 0/3 to 3/3 fired. - design-review: mean delta +0.11 -> +0.08. 04-clean-go lost its uplift (1.00/0.67 -> 0.50/0.50); flagged as a with-arm regression to confirm. Measured costs replace the unmeasured estimates in both Run sections. Two ceiling corrections follow from the same data: 01-rate-limiter drops back to timeout_seconds 300 now that the #6 fix stops it dispatching the clean-room agent (it peaks at 164s), and 04-retry-rfc gets its own row because firing correctly now puts it over the old $0.60 / 5-turn ceiling.
codybrom
added a commit
that referenced
this pull request
Oct 8, 2026
Plugin: the skill trigger and design-it-twice fixes from #10 and #11. Site: the MCP server at clairvoyance.fyi/mcp (listed in the official MCP Registry), /install and agent setup, and agent discovery. See CHANGELOG.md. Also links the server's registry entry from /mcp and the README, and describes the server as "inspired by" the book.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4, fixes #5, fixes #6.
What changed
Trigger ownership for diff and PR review (#4) —
code-evolutionnow claims "review a diff or PR (for example before merging it)".red-flagsanddesign-reviewdefer to it, anddesign-reviewkeeps the "comprehensive or prioritized design assessment" phrasing that its own eval case depends on.design-it-twice triggers (#5) — the description now covers "how should I structure X" and RFC, design-doc and ADR prompts.
Clean-room dispatch scope (#6) — isolation mode now applies only to a design the user or the codebase supplied, not to one the model drafted itself on a fresh problem. A new convergence check in The Procedure falls back to the pre-mortem when two designs match on interface shape, state ownership and decomposition.
Equal-depth alternatives in design docs — a design doc or RFC must develop at least two alternatives to the same depth before choosing, rather than appending one-paragraph rejections.
Eval fix —
01-rate-limitertimeout raised from 300 s to 600 s. Runs that dispatch the clean-room agent take about 200 s, and one earlier run was cut off mid-answer, which looked like a skill failure.Results
Full 3-run suites with ablation, Opus agent, Sonnet judge. Δ is the with-plugin score minus without.
03-diff-review03-undo-redo04-retry-rfc01-rate-limiter02-pr-diffSuite means: red-flags +0.13 → +0.20, design-it-twice +0.18 → +0.36, design-review +0.06 → +0.08.
Traces confirm the #6 behavior: on the fresh rate-limiter problem the clean-room agent was not dispatched in any run, while
05-notif-prefs-canary, where the user supplies a design, still dispatched it in all 3 runs and scored 1.00.Every should-not-fire case stayed at 0 Skill calls:
06-neg-keyerror,06-neg-pagination,07-neg-rename,06-neg-rename.Known gaps
long-param-listfails in 2 of 3 runs on03-diff-review. The skill that now owns diff review doesn't reliably flag a new flag parameter on an already long signature as a design problem.no-strawmanfails once each on02-plugin-apiand04-retry-rfc.