Skip to content

Adversarial suite: gate cases for a design flag with no reason, an eval case lost at the same score, and deletes-data words (follow-up to #957) #1018

Closed

Description

@jayden1711

Follow-up from #957 (adversarial suite cases, part of #707): its "Follow-ups (listed, not done)" and the second review issuecomment-6333173487 (round 3, head fd9893c).

The reviewer wrote one-line mutants of the built engine, applied in memory, and the adversarial suite stayed green under each (measured by the reviewer). These are gate cases (the suite's non-flow decide()) that would kill them.

  • Q1, hold.doubt: an evaluator design flag with no reason. Add a gate case where the evaluator flags a design change with no design_reason (measured survivor, issuecomment-6333173487).
  • Q2, Q3, hold.eval: an eval outcome dropped at the same score, visible and held-out. Every suite case with inputs.evals: dropped also lowers the score, so no case separates "a case went pass to fail" from "the score went down" (measured, issuecomment-6333173487). On main, dropped is also set when a case is lost at the same score (inferred from the reviewer's reading of skilleval.ts).
  • Q11, hold.high-risk (deletes-data): the word list. Add a case with a TRUNCATE line (and DROP COLUMN / DROP SCHEMA) (measured survivor for TRUNCATE; the other two are the reviewer's suggestion, not run).

Fix by design (issuecomment-6333173487):

  • Q2/Q3: let the suite's dropped input keep the score, so the case drives the dropped-outcome path alone.
  • Q11: either add the cases, or record the deletes-data word list as a named limit, as is done for weakening.

After these land, #957's coverage count (at most 8 of 49 guarantees, with these named as open limits) can be recounted (inferred).

Activity

  1. added a commit that references this issue on Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions