/probe-resume — one-drop real-résumé sweep + corpus-match engine ("does a fixture already reproduce this defect?")

[s-annam]

Summary

Add /probe-resume — a one-drop, read-only parser sweep over a real résumé — plus the corpus-match engine that answers the question the existing probes cannot:

This résumé exposes a defect. Does any fixture in tests/fixtures/pdfs/ already reproduce it?

Today the answer lives in a maintainer's head. The result is that every real-résumé finding either (a) mints a fixture that duplicates one we already have, or (b) keeps the real résumé in the loop as the de-facto reproducer, which is exactly what our PII policy forbids. This issue makes the coverage question mechanically answerable, and it does so entirely with PII-free code over public fixtures.

The gap, concretely

We have six read-only parser probes, all committed to this repo and all designed to be pointed at a real, PII-bearing résumé:

Skill	Harness	Env var
/probe-contact	src/lib/heuristics/probe-contact.test.ts	RL_CONTACT_PDF
/probe-skills	src/lib/heuristics/probe-skills.test.ts	RL_SKILLS_PDF
/probe-experience	src/lib/heuristics/probe-experience.test.ts	RL_EXPERIENCE_PDF
/probe-education	src/lib/heuristics/probe-education.test.ts	RL_EDUCATION_PDF
/probe-achievements	src/lib/heuristics/probe-achievements.test.ts	RL_ACHIEVEMENTS_PDF
/probe-roundtrip	src/lib/heuristics/probe-roundtrip.test.ts	RL_ROUNDTRIP_PDF

Each localizes one section's defect to one parser layer. None of them answers "is this defect new?" — and running all six by hand across one résumé, then holding six reports plus a 45-fixture corpus in your head, is the ergonomic wall we keep hitting.

The key insight — we already have the equivalence key

src/lib/heuristics/repro-artifact.ts (shipped for #245) already builds a ReproArtifact: a structure-only fingerprint of a parse. Read the file header before touching anything here — it is PII-free by construction, not by filtering. The exported type admits only numbers, booleans, fixed enums, and arrays of those. There is deliberately no free-form string slot a résumé value could occupy, and repro-artifact.test.ts asserts it.

Its axes (see ReproArtifact at src/lib/heuristics/repro-artifact.ts:70):

• triggers: LayoutTrigger[] — active layout probes
• sectionSource: "markdown" | "regex" — which splitter cut the document
• pageCount, rawCharCount, extractedCharCount — density signals
• sections: { name, lineCount }[] — where the parser cut the document
• parsedCounts — hasFullName / hasEmail / hasPhone / hasLocation / hasSummary booleans + experienceCount / educationCount / skillsCount
• linkAnnotationCount
• disagreements: { kind, field, likelyCause? }[]

That makes "does fixture X reproduce résumé R's defect?" answerable by diffing two ReproArtifacts on the axes the defect actually lives on — no eyeballing, no trust, no PII. The hard half is already built. This issue is the other half.

The hole this issue must close — structural vs. value-level defects

ReproArtifact is structural, so it sees some defect classes and is blind to others. This is the real design work, and getting it wrong makes the whole thing worse than useless (a fixture that quietly parses correctly while the real résumé fails makes a green test look like a fix).

Visible to ReproArtifact — an unrecognized skills header (#282-class) shows up as parsedCounts.skillsCount: 0 plus a missing skills entry in sections. Both axes are in the artifact. A merged experience role shows up as a low experienceCount against a large experience section lineCount.

Invisible to ReproArtifact — anything where the field is present but wrong: a phone parsed with mangled digits, a name that absorbed a title, and all of /probe-roundtrip's domain (values corrupted across the parse → export → parse cycle). hasPhone: true before and after tells you nothing about whether the digits survived.

The fix is NOT to widen ReproArtifact with a string field. That breaks repro-artifact.test.ts's PII assertion and defeats the file header's entire contract. Instead, value-level classes are expressed as PII-free derived booleans, computed in memory from the values and then discarded:

// Computed from the real values, which never leave the function.
// Only the boolean is ever returned, printed, or written.
{ emailChangedAcrossRoundtrip: true, phoneDigitsChangedAcrossRoundtrip: false }

A boolean cannot leak an email. The predicate reads the values; the artifact records only the verdict.

Design

1. src/lib/heuristics/defect-classes.ts (new)

The defect-class → load-bearing-axis table. One entry per defect class the probes can emit. Each entry names the class, the probe that detects it, and the predicate that decides whether a candidate artifact carries the same defect.

export type DefectClass =
  | "skills-header-unrecognized"
  | "skills-extraction-miss"
  | "experience-roles-merged"
  | "experience-section-dropped"
  | "education-entries-under-chunked"
  | "contact-field-dropped"
  | "roundtrip-value-corrupted"
  // …one per class the six probes can localize

export interface DefectSpec {
  class: DefectClass;
  probe: "probe-contact" | "probe-skills" | /* … */ "probe-roundtrip";
  /** Which ReproArtifact axes are load-bearing for THIS class. Divergence on
   *  any other axis (e.g. pageCount) is informational, never disqualifying. */
  loadBearingAxes: readonly AxisPath[];   // e.g. ["sections.skills", "parsedCounts.skillsCount"]
  /** True when this artifact exhibits the defect. Structural classes read the
   *  artifact only. Value-level classes ALSO take a derived-boolean bag, which
   *  the probe computes in memory from the values and never persists. */
  exhibits(a: ReproArtifact, derived: DerivedSignals): boolean;
}

DerivedSignals is the escape hatch for value-level classes — a flat bag of booleans (emailChangedAcrossRoundtrip, phoneDigitsChangedAcrossRoundtrip, …). It is a fixed, boolean-only interface, mirroring ReproArtifact's discipline: no string slot, so it inherits the same type-level PII guarantee. Add an assertion test alongside repro-artifact.test.ts that pins this.

2. src/lib/heuristics/fixture-match.ts (new — the engine)

export interface FixtureCoverage {
  class: DefectClass;
  /** Fixtures whose ReproArtifact exhibits the SAME defect on the load-bearing axes. */
  coveredBy: string[];        // e.g. ["tests/fixtures/pdfs/word/skills-glyph-header.pdf"]
  /** Axes that diverged on the closest non-matching fixture — the "why not" signal. */
  nearMisses: { fixture: string; divergedAxes: AxisPath[] }[];
}

export function matchCorpus(
  real: ReproArtifact,
  realDerived: DerivedSignals,
  defects: readonly DefectClass[],
  corpus: readonly { path: string; artifact: ReproArtifact; derived: DerivedSignals }[],
): FixtureCoverage[];

Pure, lib-layer, no I/O — same discipline as repro-artifact.ts. Fully unit-testable with hand-built artifacts, no PDFs required.

3. Bake reproArtifact into the corpus snapshots

The engine needs a ReproArtifact per fixture. Rather than re-parsing 45 PDFs on every probe run, bake it: bump *.expected.json schemaVersion 4 → 5 and add a reproArtifact block (plus the fixture's derived booleans). Both are PII-free by type, so this preserves the snapshots' existing "lossy by design, never field values" property — the reason they are safe to commit at all.

npm run bake-fixtures (UPDATE_FIXTURES=1 vitest run src/lib/heuristics/corpus.test.ts) regenerates all 45. The bake is the only place a fixture is parsed for this purpose; fixture-match.ts reads the baked artifacts.

4. src/lib/heuristics/probe-resume.test.ts (new harness)

Follows the six siblings exactly — a vitest harness gated on an env var, inert (skipped) when unset, so CI never runs it. There is no standalone script and you should not write one: the pdfjs worker uses Vite's ?url import, which resolves only under the Vite/vitest transform, so plain tsx/node breaks. The vitest run is the execution vehicle.

RL_RESUME_PDF=/abs/path/to/real-resume.pdf \
  npx vitest run src/lib/heuristics/probe-resume.test.ts

Var	Default	Meaning
RL_RESUME_PDF	(unset)	Absolute path to the résumé PDF. Unset → harness is inert.
RL_RESUME_OUT	internal/resume/	Directory for the full JSON report. Default lives under internal/, which is gitignored (.gitignore:61). Override only with another gitignored/out-of-repo path.

It runs one runCascade(), drives each section probe's existing localization logic against that single parse (do not re-parse six times), computes the derived booleans (including the round-trip re-parse), builds the ReproArtifact, and calls matchCorpus.

5. .claude/skills/probe-resume/SKILL.md (new)

The skill doc, carrying the same PII guardrail block as its six siblings, verbatim in spirit:

1. The input PDF is local-only. NEVER commit it. It is not a fixture. tests/fixtures/pdfs/ is synthetic-personas-only by policy (tests/fixtures/pdfs/README.md).
2. Values → gitignored scratch only. Console prints counts, classes, and coverage — never a candidate's real values. Cite a defect by class, never by value.

The skill is read-only and commits nothing. It ends by printing, for each uncovered defect, the exact next-step invocation to mint a fixture — it does not mint one itself.

The output — the coverage map

The whole point, printed at the end of the run:

DEFECTS FOUND (3)

  skills-header-unrecognized     → COVERED  tests/fixtures/pdfs/word/skills-glyph-header.pdf
  experience-roles-merged        → COVERED  tests/fixtures/pdfs/latex/two-column-roles.pdf
  roundtrip-value-corrupted      → NO FIXTURE COVERS THIS
        nearest: tests/fixtures/pdfs/word/basic.pdf (diverged: derived.emailChangedAcrossRoundtrip)
        → mint a synthetic fixture that reproduces this class, then add a *.repro.test.ts pinning it

COVERAGE  2/3 defects already pinned by the corpus

COVERED means: stop. The corpus already reproduces this; go fix the parser against the existing fixture and never open the real résumé again. NO FIXTURE COVERS THIS is the only case that justifies minting a new one.

Reuse analysis

Capability — sweep every parser section over one résumé and report which corpus fixtures already reproduce the defects found.

Existing surfaces found:

• .claude/skills/probe-{contact,skills,experience,education,achievements,roundtrip}/SKILL.md + their src/lib/heuristics/probe-*.test.ts harnesses — each owns one section, read-only, real-résumé input, gitignored output.
• src/lib/heuristics/repro-artifact.ts — the PII-free parse fingerprint.
• src/lib/heuristics/corpus.test.ts + *.expected.json — the baked, PII-free corpus snapshots.
• *.repro.test.ts (section-routing.repro.test.ts, page-furniture-experience.repro.test.ts, two-column-experience-values.repro.test.ts) — the existing convention for pinning one defect to one fixture.

Decision — build new, and extend the rest. No existing surface owns cross-section sweep or corpus coverage; each probe is deliberately single-section, and widening one of them (say /probe-skills) to cover all six sections would break the family's one-probe-one-layer contract that makes them readable. But /probe-resume is an orchestrator, not a reimplementation: it must reuse the six harnesses' localization logic and buildReproArtifact() as-is. If the six probes' logic isn't importable today, extract it into shared functions and have both the probe and the sweep call it — do not copy-paste six detectors into a seventh file. ReproArtifact and the corpus snapshots are extended (a baked block, a bumped schemaVersion), not forked.

Implementation plan

1. src/lib/heuristics/defect-classes.ts — DefectClass union, DerivedSignals (boolean-only), DefectSpec, and the table. One entry per class the six probes localize today. Start from each probe's existing verdict strings (HEADER-UNRECOGNIZED, EXTRACTION-MISS, NO-SKILLS-SECTION, …) — they are already the class taxonomy, just unnamed.
2. src/lib/heuristics/defect-classes.test.ts — pin the PII contract: DerivedSignals admits no string. Mirror the assertion in repro-artifact.test.ts.
3. src/lib/heuristics/fixture-match.ts — matchCorpus(). Pure. Plus fixture-match.test.ts with hand-built artifacts (no PDF I/O).
4. Extract shared localization from the six probe-*.test.ts harnesses into importable functions (each returning DefectClass[] for a given parse), so the sweep reuses rather than duplicates. Keep each probe's console output byte-identical — this is a refactor, not a behavior change.
5. Bump *.expected.json to schemaVersion: 5 — add the reproArtifact + derived blocks in corpus.test.ts's bake path; run npm run bake-fixtures to regenerate all 45.
6. src/lib/heuristics/probe-resume.test.ts — the harness: one runCascade(), run all six localizers, compute derived booleans (incl. the export → re-parse hop for round-trip classes), build the artifact, call matchCorpus(), print the coverage map, mirror the full JSON to RL_RESUME_OUT.
7. .claude/skills/probe-resume/SKILL.md — the skill doc, with the PII guardrail and the "prints the next step, mints nothing" contract. Add it to the probe-* sibling cross-links in the other six SKILL.md frontmatters.

Acceptance criteria

• RL_RESUME_PDF=/abs/path npx vitest run src/lib/heuristics/probe-resume.test.ts sweeps all six sections from a single runCascade() and prints one consolidated defect list.
• With RL_RESUME_PDF unset, the harness is inert (skipped) — npm run verify is unaffected and CI never runs it.
• For each defect found, the run prints COVERED <fixture path> or NO FIXTURE COVERS THIS + the nearest fixture and the axes that diverged.
• matchCorpus() is pure and lib-layer (no I/O), and fixture-match.test.ts covers it with hand-built artifacts — no PDF required.
• The defect-class table has an entry for every verdict the six existing probes can emit today; a probe verdict with no table entry is a test failure, not a silent gap.
• A value-level defect is detectable. A round-trip corruption (/probe-roundtrip's domain) is matched via a DerivedSignals boolean — proven by a test in which two fixtures with identical ReproArtifacts are correctly distinguished.
• ReproArtifact gains no string field. repro-artifact.test.ts's PII assertion still passes, unmodified. DerivedSignals carries the same boolean-only assertion.
• *.expected.json is at schemaVersion: 5 with a baked reproArtifact; all 45 fixtures regenerated via npm run bake-fixtures; the snapshots still contain no field values (verified by grep, not by assertion).
• The six existing probe-* harnesses still print byte-identical output after the shared-localizer extraction.
• .claude/skills/probe-resume/SKILL.md exists, carries the PII guardrail, states that it commits nothing, and is cross-linked from the six sibling SKILL.md files.
• npm run verify green (typecheck → lint → coverage → build → fallow).

Out of scope

• Minting the synthetic fixture. Re-exporting a template with a synthetic persona is still a human step (tests/fixtures/pdfs/README.md). /probe-resume tells you a fixture is needed and prints the next-step command; it does not create one.
• Committing anything. The probe is read-only. Fixture PRs are a separate, human-reviewed act.
• The private derivation record that maps a real résumé to the fixture standing in for it. That is tracked separately, outside this repo, for PII reasons — it consumes the engine built here, so nothing in this issue depends on it and it must not be referenced from the code or the skill doc.

Notes

• Read src/lib/heuristics/repro-artifact.ts's file header before writing any code here. It explains, at length, why "helpfully" adding a sample line or the failing bullet to the artifact is the one change that breaks everything. The same reasoning governs DerivedSignals.
• The pdfjs ?url worker constraint (vite-node/vitest only, never plain tsx/node) is documented in all six existing probe SKILL.md files. It applies here unchanged.

[s-annam]

AC addition — the out-dir must be proven gitignored, not assumed

The gap

This issue defaults RL_RESUME_OUT to internal/resume/, on the reasoning that internal/ is gitignored here (.gitignore:61). That is true of this repo — and this repo is the wrong one to check.

internal/ is a mount point for a separate, private working tree. A path under it is governed by that tree's ignore rules, not ours, and those rules enumerate the probe output dirs one by one rather than ignoring the parent. A seventh probe therefore does not inherit the six siblings' protection: probe-resume's report — which, like theirs, carries the real name / email / phone so a parse corruption is visible — would land in a directory that is tracked by default over there.

That is now fixed on the other side, so the documented default is safe before this issue is implemented.

Why that is not sufficient, and what changes here

An ignore rule defends the default path. It defends nothing about an override — RL_RESUME_OUT=~/Desktop/out, or a path inside this repo, is one env var away and silently writes real field values to a tracked location. The six existing probes carry the same latent hole; probe-resume is the one to close it, because it is the one being built.

Adding an acceptance criterion:

• The harness refuses to write when its output directory is not gitignored. Before writing the report, resolve RL_RESUME_OUT and check it against the ignore rules of whatever working tree actually contains it — git check-ignore -q <dir> executed from that directory, so the check follows the containing tree rather than this one. Not ignored → fail the run with a message naming the directory and what to add. No report, no partial write, no warn-and-continue.

That inverts the guarantee from "the documented default happens to be safe" to "an unsafe destination is not writable," which is the property the PII policy actually needs. It is a few lines, and it is the natural place to later cover the six sibling probes too.

Nothing else in the issue changes; the engine, the defect-class table, and the schema bump are unaffected.