Skip to content

Add aggregate AI structure signals #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
theclaymethod wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add aggregate AI structure signals #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 150 additions & 0 deletions evals/adversarial-evals.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12933,6 +12933,156 @@
}
]
},
{
"id": "STRUCT-18",
"category": "scanner_false_negative",
"title": "Repeated rhetorical triads are an aggregate AI cadence tell",
"target": "script",
"command": [
"python3",
"scripts/structure_scan.py"
],
"stdin": "The rollout promised faster decisions, clearer ownership, and stronger accountability. The memo called for consistent planning, disciplined execution, and transparent reporting. Leaders said the change would produce durable growth, meaningful alignment, and lasting momentum. Yet the memo named no changed process, deadline, or owner.",
"failure_mode": "structure_scan records triad_density but never flags it, so prose built from repeated rules of three passes clean.",
"correct_behavior": "Exit 1 and report triad_density after three rhetorical triads in a short prose passage.",
"assertions": [
{
"type": "exit_code",
"equals": 1
},
{
"type": "json",
"path": "flagged.triad_density",
"equals": true
}
]
},
{
"id": "STRUCT-19",
"category": "scanner_false_positive",
"title": "Technical inventory lists may repeat factual triads",
"target": "script",
"command": [
"python3",
"scripts/structure_scan.py",
"--genre",
"docs"
],
"stdin": "The kit contains a wrench, gauge, and seal. The service form records unit, date, and technician. The shipment includes pump, hose, and bracket. Record each factual field exactly as shown.",
"failure_mode": "A blanket rule-of-three detector would punish compact factual enumerations in reference documentation.",
"correct_behavior": "Exit 0 under --genre docs because factual inventories legitimately use repeated three-item lists.",
"assertions": [
{
"type": "exit_code",
"equals": 0
},
{
"type": "json",
"path": "flags",
"equals": []
}
]
},
{
"id": "STRUCT-20",
"category": "scanner_false_negative",
"title": "Long sentences repeatedly chained with and are an aggregate AI tell",
"target": "script",
"command": [
"python3",
"scripts/structure_scan.py"
],
"stdin": "Each morning, the platform reviews every request and compares the account history and estimates the likely outcome before the assigned team receives a recommendation. After lunch, managers inspect the recommendation and discuss the surrounding context and approve the next action after a final regional review. Before noon, the workflow records each decision and updates the central dashboard and sends a notice to every regional owner with an open account. During review, analysts monitor the dashboard and compare weekly patterns and prepare a narrative summary for the monthly operating meeting with senior managers. On Fridays, directors read the summary and request additional context and circulate the revised account to the wider leadership group for another discussion. Without resolution, the process repeats each week and creates another layer of commentary and leaves the underlying ownership question unanswered by any named manager.",
"failure_mode": "Long average sentence length alone misses the specific LLM habit of repeatedly chaining clauses with 'and'.",
"correct_behavior": "Exit 1 and report and_heavy_long_sentence_share when at least half of six sentences are long and contain multiple uses of 'and'.",
"assertions": [
{
"type": "exit_code",
"equals": 1
},
{
"type": "json",
"path": "flagged.and_heavy_long_sentence_share",
"equals": true
}
]
},
{
"id": "STRUCT-21",
"category": "scanner_false_positive",
"title": "Procedural documentation can legitimately chain required operations",
"target": "script",
"command": [
"python3",
"scripts/structure_scan.py",
"--genre",
"docs"
],
"stdin": "The controller reads the header and validates the checksum and rejects the frame when either field is invalid. Next the worker opens the archive and verifies the manifest and copies the approved entries into the staging directory. A client signs the request and attaches the timestamp and sends both values in the authorization header. Each service parses the token and checks the audience and returns status 401 when the audience does not match. Before backup the job locks the database and writes the snapshot and releases the lock only after the checksum succeeds. During restore the command reads the snapshot and rebuilds the index and prints the final record count for comparison.",
"failure_mode": "A prose-wide conjunction rule would flag accurate step semantics in technical reference material.",
"correct_behavior": "Exit 0 under --genre docs because the repeated conjunctions preserve ordered technical operations.",
"assertions": [
{
"type": "exit_code",
"equals": 0
},
{
"type": "json",
"path": "flags",
"equals": []
}
]
},
{
"id": "STRUCT-22",
"category": "scanner_false_negative",
"title": "Long punctuation-sparse prose is an aggregate AI tell",
"target": "script",
"command": [
"python3",
"scripts/structure_scan.py"
],
"stdin": "The organization launched the program after several planning sessions that produced a broad mandate without assigning a specific owner to the daily work. Senior leaders described the effort as a durable response to changing market conditions while leaving each regional team to interpret the operating model alone. Employees received a lengthy explanation that connected customer expectations with internal priorities without identifying which existing commitments would stop. The first quarterly review described encouraging movement across several workstreams although the report did not define the baseline used for that judgment. A later update repeated the original ambition in different language while offering no dates for the systems that supposedly supported the transition. By the end of the year the initiative had generated many polished summaries without producing a decision that frontline managers could apply during an actual customer call.",
"failure_mode": "Current metrics can record long sentences but do not catch a sustained absence of commas, semicolons, colons, parentheses, or dashes inside long prose.",
"correct_behavior": "Exit 1 and report sparse_internal_punctuation when 110-plus words of long-sentence prose contain almost no internal punctuation.",
"assertions": [
{
"type": "exit_code",
"equals": 1
},
{
"type": "json",
"path": "flagged.sparse_internal_punctuation",
"equals": true
}
]
},
{
"id": "STRUCT-23",
"category": "scanner_false_positive",
"title": "Long reference prose may use few internal punctuation marks",
"target": "script",
"command": [
"python3",
"scripts/structure_scan.py",
"--genre",
"docs"
],
"stdin": "The command reads the selected configuration from the workspace before it creates any output in the destination directory chosen by the operator. Each source record passes through the validation stage that checks required fields before the converter writes the normalized representation to disk. The program stops before conversion when a required identifier is absent because downstream tools cannot reconstruct that value from the remaining fields. Operators can rerun the command after correcting the source file because the failed attempt does not alter the original input or the destination. The final verification step reads every generated record from disk before the command reports success to the calling process. This behavior lets automated jobs treat a zero exit status as evidence that every generated record was readable at the end of the run.",
"failure_mode": "A punctuation-sparsity threshold without a genre guard would reject valid reference prose written as complete declarative sentences.",
"correct_behavior": "Exit 0 under --genre docs because sparse punctuation is not independently suspicious in technical reference material.",
"assertions": [
{
"type": "exit_code",
"equals": 0
},
{
"type": "json",
"path": "flags",
"equals": []
}
]
},
{
"id": "SKILL-TITLE-01",
"category": "headline_reconstruction",
Expand Down
6 changes: 3 additions & 3 deletions references/packs/pack-structure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ Use this pack for document-level AI structure. Do not report isolated phrases un

## Look For

- `structure_scan.py` flags: `sentence_burstiness`, `conclusion_coda`, `bold_colon_listicle`, `one_line_staccato`, `connective_paragraph_openers`, `signpost_density`, `opener_repetition`, `participial_closer_share`.
- Concrete thresholds to self-check: average sentence length under 8 or over 34 words is suspicious; sentence-length variance under 18 on 5+ sentences suggests uniform rhythm; 3+ one-line paragraphs under 8 words suggests staccato; 3+ consecutive paragraphs opened by connectives/signposts suggests scaffolding; 4+ repeated paragraph openers is blocking; 35%+ participial sentence endings suggests templated cadence.
- `structure_scan.py` flags: `sentence_burstiness`, `conclusion_coda`, `bold_colon_listicle`, `one_line_staccato`, `connective_paragraph_openers`, `signpost_density`, `opener_repetition`, `participial_closer_share`, `triad_density`, `and_heavy_long_sentence_share`, `sparse_internal_punctuation`.
- Concrete thresholds to self-check: average sentence length under 8 or over 34 words is suspicious; sentence-length variance under 18 on 5+ sentences suggests uniform rhythm; 3+ one-line paragraphs under 8 words suggests staccato; 3+ consecutive paragraphs opened by connectives/signposts suggests scaffolding; 4+ repeated paragraph openers is blocking; 35%+ participial sentence endings suggests templated cadence; 3+ compact triads in 40+ words suggests rule-of-three packaging; half of 6+ sentences being 20+ words with 2+ uses of "and" suggests clause chaining; fewer than 1 internal punctuation mark per 100 words is suspicious only when 110+ words average 19+ words per sentence.
- Judge-only macro tells from `taboo-phrases.md`: both-sidesism, templated redemption arc, preview/recap symmetry, over-determination, uniform emotional register.
- `silhouette_scan.py` flags idea arrangement above the surface: `scaffold_opener_share`, `callback_content` (recap loop), `role_entropy_bits`, `preview_fulfillment`, `heading_preview`, composited into `silhouette_penalty` (flags at `>= 1.0`).
- A high `silhouette_penalty` means the document follows a template outline (preview-then-fulfill, rotating scaffold openers, closing recap); cite the specific flagged metric, not just the composite.
- Cue-deletion evades silhouette but not surface repetition: pair `silhouette_scan` with `structure_scan` (`opener_repetition`, `one_line_staccato`) so a stripped-connective attack still fails one of the two fences.

Genre matters. `--genre docs` can allow reference-doc structure; `--genre social` can allow deliberate social cadence. Output still fails if the genre excuse is false.
Genre matters. `--genre docs` allows factual triads, conjunction-chained procedures, and sparse punctuation in reference prose; `--genre social` can allow deliberate social cadence. Output still fails if the genre excuse is false.

## Emit

Expand Down
9 changes: 8 additions & 1 deletion references/taboo-phrases.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1026,10 +1026,17 @@ scanner-enforced.
| Over-determination | Agent judgment — not scanner-enforced. Watch for explicit theme-stating that explains what the reader should infer. | Explainers, tutorials, and accessibility-minded docs may need explicitness. |
| Elegant variation pointer | Agent judgment — not scanner-enforced. Watch for one referent cycling through many labels to avoid repetition. | Literary prose and journalism may vary references for rhythm, as long as clarity survives. |
| Uniform emotional register | Agent judgment — not scanner-enforced. Watch for every paragraph landing at the same polished confidence level. | Formal reports and reference docs may intentionally keep emotional texture low. |
| Repeated rhetorical triads | Scanner-enforced (`triad_density`) only after three compact Oxford-comma triads in 40+ prose words. Preserve factual lists. | Reference docs and inventories legitimately repeat three-item enumerations; use `--genre docs`. |
| Conjunction-heavy long sentences | Scanner-enforced (`and_heavy_long_sentence_share`) when at least half of six sentences are 20+ words with two or more uses of "and". | Ordered technical operations may require the conjunctions; use `--genre docs`. |
| Sparse internal punctuation in long prose | Scanner-enforced (`sparse_internal_punctuation`) only over 110+ words when mean sentence length is at least 19 and internal marks fall below one per 100 words. | Complete declarative reference prose can be intentionally sparse; use `--genre docs`. |

Related deterministic metrics: `sentence_burstiness`, `bold_colon_listicle`,
`one_line_staccato`, `connective_paragraph_openers`, `signpost_density`,
`opener_repetition`, and `participial_closer_share`.
`opener_repetition`, `participial_closer_share`, `triad_density`,
`and_heavy_long_sentence_share`, and `sparse_internal_punctuation`. The three
aggregate cadence checks above follow the corpus findings in The Economist's
2026 study, ["How to spot AI writing"](https://archive.ph/i9UQP); they do not
turn the study's individual example words into blanket bans.

### Silhouette (discourse-level)

Expand Down
31 changes: 26 additions & 5 deletions scripts/banned_phrase_scan.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,16 +104,25 @@ def _line_starts(text: str) -> list[int]:
return starts


def _line_col_context(text: str, line_starts: list[int], pos: int) -> tuple[int, int, str]:
def _line_col_context(
text: str,
line_starts: list[int],
pos: int,
context_cache: dict[int, str] | None = None,
) -> tuple[int, int, str]:
"""Derive (1-based line, 1-based column, stripped line context) for an offset
into ORIGINAL text from a precomputed line_starts index, in O(log n)."""
idx = bisect.bisect_right(line_starts, pos) - 1
line_start = line_starts[idx]
line_end = line_starts[idx + 1] - 1 if idx + 1 < len(line_starts) else len(text)
line_num = idx + 1
column = pos - line_start + 1
context = text[line_start:line_end].strip()
context = context[:100] + "..." if len(context) > 100 else context
context = context_cache.get(idx) if context_cache is not None else None
if context is None:
context = text[line_start:line_end].strip()
context = context[:100] + "..." if len(context) > 100 else context
if context_cache is not None:
context_cache[idx] = context
return line_num, column, context


Expand Down Expand Up @@ -509,6 +518,13 @@ def _line_col_context(text: str, line_starts: list[int], pos: int) -> tuple[int,
}

# Structural patterns (regex)
# Compile literal phrase regexes once at module load. In-process callers such as
# harvest and refine time scan_for_violations itself and may scan many documents;
# they should not pay the one-time compilation cost on the first document.
for _banned_phrase in BANNED_PHRASES:
_phrase_pattern_ci(_banned_phrase)


STRUCTURAL_PATTERNS: list[dict[str, str]] = [
# Standalone "Period." / "Full stop." for emphasis — only when it stands as
# its own sentence, not when a clause merely ends in the word "period".
Expand Down Expand Up @@ -1162,6 +1178,7 @@ def scan_for_violations(text: str, include_quoted: bool = False) -> list[Violati
spans: list[tuple[int, int]] = []
scan_text = mask_ignored_spans(text, include_quoted=include_quoted)
line_starts = _line_starts(text)
line_context_cache: dict[int, str] = {}

# Check banned phrases. Matching runs case-insensitively directly on
# scan_text (original case, masking is length-preserving) instead of on a
Expand All @@ -1170,7 +1187,9 @@ def scan_for_violations(text: str, include_quoted: bool = False) -> list[Violati
for phrase, info in BANNED_PHRASES.items():
for match in _phrase_pattern_ci(phrase).finditer(scan_text):
pos = match.start()
line_num, column, context = _line_col_context(text, line_starts, pos)
line_num, column, context = _line_col_context(
text, line_starts, pos, line_context_cache
)

violations.append({
"phrase": phrase,
Expand All @@ -1191,7 +1210,9 @@ def scan_for_violations(text: str, include_quoted: bool = False) -> list[Violati
continue
for match in matches:
pos = match.start()
line_num, column, context = _line_col_context(text, line_starts, pos)
line_num, column, context = _line_col_context(
text, line_starts, pos, line_context_cache
)

violations.append({
# Preserve the pre-fix lowercase phrase field: STRUCTURAL_PATTERNS
Expand Down
Loading
Loading