You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Adversarial suite: gate cases for a design flag with no reason, an eval case lost at the same score, and deletes-data words (follow-up to #957) #1018
Follow-up from #957 (adversarial suite cases, part of #707): its "Follow-ups (listed, not done)" and the second review issuecomment-6333173487 (round 3, head fd9893c).
The reviewer wrote one-line mutants of the built engine, applied in memory, and the adversarial suite stayed green under each (measured by the reviewer). These are gate cases (the suite's non-flow decide()) that would kill them.
Q1, hold.doubt: an evaluator design flag with no reason. Add a gate case where the evaluator flags a design change with no design_reason (measured survivor, issuecomment-6333173487).
Q2, Q3, hold.eval: an eval outcome dropped at the same score, visible and held-out. Every suite case with inputs.evals: dropped also lowers the score, so no case separates "a case went pass to fail" from "the score went down" (measured, issuecomment-6333173487). On main, dropped is also set when a case is lost at the same score (inferred from the reviewer's reading of skilleval.ts).
Q11, hold.high-risk (deletes-data): the word list. Add a case with a TRUNCATE line (and DROP COLUMN / DROP SCHEMA) (measured survivor for TRUNCATE; the other two are the reviewer's suggestion, not run).
Fix by design (issuecomment-6333173487):
Q2/Q3: let the suite's dropped input keep the score, so the case drives the dropped-outcome path alone.
Q11: either add the cases, or record the deletes-data word list as a named limit, as is done for weakening.
After these land, #957's coverage count (at most 8 of 49 guarantees, with these named as open limits) can be recounted (inferred).
Activity
Merge pull request #1048 from jayden1711/issue-1018-adversarial-gate-…