Skip to content

extract_constraints: naive backtick and quote pairing makes validate_preservation false-fail on prose-only edits #8

Description

@thebristolsound

Summary

scripts/extract_constraints.py pairs backticks and quote marks positionally,
with no awareness of fenced blocks or paragraph boundaries. Both patterns can
capture ordinary prose as a code or quote constraint. Because
validate_preservation.py treats those constraints as must-preserve, the
blocking gate then reports passed: false for a rewrite that changed only
prose — the exact edit the contract asks for.

Found while running /unslop rewrite over four Markdown/MDX whitepapers
(~18k words). Two of the four false-failed. Every atomic constraint (numbers,
paths, dates, names) survived; a hand-rolled constraint diff confirmed it.

Repro 1 — fenced block, extract_constraints.py:71

The code pattern cannot match a three-backtick fence. It matches from the
opener's third backtick to the closer's first, leaving two stray backticks that
pair with the next inline span.

Input:

Install with `pnpm install` before anything else.

```bash
pnpm build
```

This paragraph is a moralizing recap coda that adds nothing.

Run `pnpm lint` last. The MHTML cap is 200 MB.

Extractor output:

  code -> '`pnpm install`'
  code -> '`bash\npnpm build\n`'
  code -> '`\n\nThis paragraph is a moralizing recap coda that adds nothing.\n\nRun `'
measurement -> '200 MB'

Delete that coda and change nothing else:

$ python3 scripts/validate_preservation.py orig.md new.md
passed: False  total: 4  missing: 1
  MISSING code -> '`\n\nThis paragraph is a moralizing recap coda that adds nothing.\n\nRun `'

Repro 2 — short quote, extract_constraints.py:74

The quote pattern requires ten or more characters between the marks, so it
skips anything shorter. That quotation's closing mark then becomes an opener and
the match runs to the next mark, across paragraphs.

Input:

Add a crossed-out box labeled "telemetry" to make it explicit.

This closing paragraph restates what you already read and should go.

The guard rejects the "strip the signatures by rewriting" attack.

Extractor output:

quote -> '" to make it explicit.\n\nThis closing paragraph restates what you already read and should go.\n\nThe guard rejects the "'

The word telemetry is nine characters inside the marks, so it never matches,
and the drift begins there. In the real document this produced a single
7,200-character quote constraint spanning six sections.

Impact

validate_preservation is the one blocking gate in the core contract's
validation battery. On Markdown or MDX carrying fences, inline code, or short
quoted terms, it fails correct rewrites and passes only when an edit happens to
fall outside a phantom span. An agent following the contract has to either
override the gate or discard a valid rewrite.

Suggested direction

  • Match fenced blocks first as their own constraint type, then run the inline
    pattern over what remains.
  • Forbid a blank line inside quote, and reconsider the ten-character floor
    that starts the drift.

Both live in PATTERNS, so the change is local to extract_constraints.py.

Environment

d81f519, Python 3.12, Linux. python3 evals/check.py is green on a clean
tree, so no existing row covers this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions