docs(testing): instrument-integrity check — a record that cannot vary is not evidence

[garthsch]

TestingDoctrine.md § The probe that fits the claim covers instrument selection: bun test on a DOM claim is the wrong instrument, and the claim "would close on evidence that doesn't actually test it." It does not cover instrument integrity — a correctly-chosen probe whose capture path is structurally incapable of carrying the signal. That path reports silence on every failure, and silence reads as pass. Nothing in the doctrine or the anti-pattern list asks whether the recorded evidence can vary.

The common shape is a truncated subprocess stderr. A child that throws emits the runtime's source preamble first, so stderr.slice(0, N) keeps the same text on every failure while the cause sits at the tail. Each decision is individually reasonable — throw on error, include stderr in the message, bound the stored string — and they compose into total diagnostic loss with no symptom.

What it cost on one install: a nightly pipeline recorded 20 consecutive failures across two nights whose stored last_error held the failing function's own TypeScript source lines instead of the error. The failures were transient, so re-running by hand succeeded and the real cause is now unrecoverable for all 20. Worse than the lost week: a status report attributed causes drawn partly from that empty channel before anyone checked whether the channel worked. The bug was in the capture, but the expensive mistake was reading its output as data.

Changes (docs only, no behavior change)

• The anti-pattern list gains one row: closing a claim on a log field, error string or exit-code message without first checking the field can vary.
• § The probe that fits the claim gains a short ### The instrument must be able to report failure subsection: the failure shape, a WRONG/RIGHT pair for the truncated-stderr case, and the check reduced to one question — would this field differ if the cause were different? Run the probe against a second, known-different failure and compare the stored text.
• The subsection draws two boundaries: mutation testing (skills/Hardening) asks whether a test would fail if the code were wrong, this asks whether a record would differ if the cause were different; and it names itself the evidence-side companion to Rule 11, where a test must produce the edge rather than hand-write the state.

Deliberately not touched

No enforcement, no hook, no runtime gate — this is doctrine prose and one anti-pattern row, consistent with the Algorithm being prompt-driven by design. Rules 1-12 are unchanged. No code, schema, settings or test surface is in the diff. Frontmatter is untouched: version, last_updated and last_updated_by look governed by pai-freshness-v1 and the release process rather than by contributors.

Related, and why this is a different problem

Three nearby threads, none of which covers this:

• docs(testing): scope the test-location rule — subsystem packages keep their suite in-package #1603 (docs(testing), same file) scoped where tests live. Structurally the same kind of contribution, different subject. Noted mainly because its resolution is the one I'm expecting here.
• Proposal: "contract probe before build" verification rule + Algorithm claim 5 extended from prerequisites to contracts #1952 → Verification rule 8, "contract probe before build", covers evidence gathered too late: the probe is real and would have varied, it just wasn't run until after the build. This covers evidence that could never have varied at all — the probe ran, on time, and recorded an artifact. Rule 8 tells you when to look; this tells you whether what you're looking at can say anything.
• fix: MemoryRetriever writes the retrieval evidence MemorySystem.md promises (#1910) #2023 / MemorySystem.md documents MemoryRetriever as the producer of memory-retrievals.jsonl, but it never writes it — CortexHealth's retrieval-missing warn can never clear #1910 covers a documented evidence file with no producer — the checker shipped without the thing that writes it, so the record is absent. Here the record is present, well-formed, and written on every failure, which is why nothing warns. Absent evidence announces itself; invariant evidence does not.

Inside the doctrine itself, § The probe that fits the claim covers picking the right instrument, and Rule 11 covers producing the edge rather than hand-writing the state. Neither asks whether the instrument's output can vary once it is running.

Verification

• Branch cut from 5e2f2e8 (upstream main at time of writing); patch applies with no rejects; diff is one file, +28 lines.
• Reaches a fresh install: LifeOS/Tools/DeployCore.ts copies install/LIFEOS/<entry> to <configRoot>/LIFEOS/, so this file is in the deployed payload, not repo-only.
• Renders: fences balanced (one added pair, no unterminated block); the new ### sits level with its siblings under the same ##; the /\r?\n/ literal survives the patch round-trip.
• Not done: a fresh-system install. The change is inert prose in an already-shipped file and the deploy-path check above is the proportionate substitute. Happy to run one if you'd rather.

Filed as a PR because the diff is the clearest statement of the proposal. Per #1603, if it's easier to take the text into source and close this, that works the same for me. And if some version of this is already sitting in source from a recent sweep, say so and I'll close it myself.

[danielmiessler]

Thanks @garthsch. The gap you named is real: the doctrine covered choosing the right probe but never asked whether the probe's record can differ between two failures. Your section is in LIFEOS/DOCUMENTATION/Testing/TestingDoctrine.md as "The instrument must be able to report failure", with the truncated-stderr example and an anti-pattern line, credited to you. Next release carries it.