Summary
scripts/extract_constraints.py pairs backticks and quote marks positionally,
with no awareness of fenced blocks or paragraph boundaries. Both patterns can
capture ordinary prose as a code or quote constraint. Because
validate_preservation.py treats those constraints as must-preserve, the
blocking gate then reports passed: false for a rewrite that changed only
prose — the exact edit the contract asks for.
Found while running /unslop rewrite over four Markdown/MDX whitepapers
(~18k words). Two of the four false-failed. Every atomic constraint (numbers,
paths, dates, names) survived; a hand-rolled constraint diff confirmed it.
Repro 1 — fenced block, extract_constraints.py:71
The code pattern cannot match a three-backtick fence. It matches from the
opener's third backtick to the closer's first, leaving two stray backticks that
pair with the next inline span.
Input:
Install with `pnpm install` before anything else.
```bash
pnpm build
```
This paragraph is a moralizing recap coda that adds nothing.
Run `pnpm lint` last. The MHTML cap is 200 MB.
Extractor output:
code -> '`pnpm install`'
code -> '`bash\npnpm build\n`'
code -> '`\n\nThis paragraph is a moralizing recap coda that adds nothing.\n\nRun `'
measurement -> '200 MB'
Delete that coda and change nothing else:
$ python3 scripts/validate_preservation.py orig.md new.md
passed: False total: 4 missing: 1
MISSING code -> '`\n\nThis paragraph is a moralizing recap coda that adds nothing.\n\nRun `'
Repro 2 — short quote, extract_constraints.py:74
The quote pattern requires ten or more characters between the marks, so it
skips anything shorter. That quotation's closing mark then becomes an opener and
the match runs to the next mark, across paragraphs.
Input:
Add a crossed-out box labeled "telemetry" to make it explicit.
This closing paragraph restates what you already read and should go.
The guard rejects the "strip the signatures by rewriting" attack.
Extractor output:
quote -> '" to make it explicit.\n\nThis closing paragraph restates what you already read and should go.\n\nThe guard rejects the "'
The word telemetry is nine characters inside the marks, so it never matches,
and the drift begins there. In the real document this produced a single
7,200-character quote constraint spanning six sections.
Impact
validate_preservation is the one blocking gate in the core contract's
validation battery. On Markdown or MDX carrying fences, inline code, or short
quoted terms, it fails correct rewrites and passes only when an edit happens to
fall outside a phantom span. An agent following the contract has to either
override the gate or discard a valid rewrite.
Suggested direction
- Match fenced blocks first as their own constraint type, then run the inline
pattern over what remains.
- Forbid a blank line inside
quote, and reconsider the ten-character floor
that starts the drift.
Both live in PATTERNS, so the change is local to extract_constraints.py.
Environment
d81f519, Python 3.12, Linux. python3 evals/check.py is green on a clean
tree, so no existing row covers this.
Summary
scripts/extract_constraints.pypairs backticks and quote marks positionally,with no awareness of fenced blocks or paragraph boundaries. Both patterns can
capture ordinary prose as a
codeorquoteconstraint. Becausevalidate_preservation.pytreats those constraints as must-preserve, theblocking gate then reports
passed: falsefor a rewrite that changed onlyprose — the exact edit the contract asks for.
Found while running
/unslop rewriteover four Markdown/MDX whitepapers(~18k words). Two of the four false-failed. Every atomic constraint (numbers,
paths, dates, names) survived; a hand-rolled constraint diff confirmed it.
Repro 1 — fenced block,
extract_constraints.py:71The
codepattern cannot match a three-backtick fence. It matches from theopener's third backtick to the closer's first, leaving two stray backticks that
pair with the next inline span.
Input:
Extractor output:
Delete that coda and change nothing else:
Repro 2 — short quote,
extract_constraints.py:74The
quotepattern requires ten or more characters between the marks, so itskips anything shorter. That quotation's closing mark then becomes an opener and
the match runs to the next mark, across paragraphs.
Input:
Extractor output:
The word
telemetryis nine characters inside the marks, so it never matches,and the drift begins there. In the real document this produced a single
7,200-character
quoteconstraint spanning six sections.Impact
validate_preservationis the one blocking gate in the core contract'svalidation battery. On Markdown or MDX carrying fences, inline code, or short
quoted terms, it fails correct rewrites and passes only when an edit happens to
fall outside a phantom span. An agent following the contract has to either
override the gate or discard a valid rewrite.
Suggested direction
pattern over what remains.
quote, and reconsider the ten-character floorthat starts the drift.
Both live in
PATTERNS, so the change is local toextract_constraints.py.Environment
d81f519, Python 3.12, Linux.python3 evals/check.pyis green on a cleantree, so no existing row covers this.