Skip to content

skill-authoring §7: bare arm is environment-relative; harvest the arms' reasoning, not just verdicts - #86

Open
firaen22 wants to merge 3 commits into
F-e-u-e-r:mainfrom
firaen22:fold-gate-r7
Open

skill-authoring §7: bare arm is environment-relative; harvest the arms' reasoning, not just verdicts#86
firaen22 wants to merge 3 commits into
F-e-u-e-r:mainfrom
firaen22:fold-gate-r7

Conversation

@firaen22

Copy link
Copy Markdown
Contributor

Summary

Two clauses folded into the existing §7 bare-probe bullet, distilled from a downstream consumer's first four fold probes of this repo's merged rules (pre-registered arms, weaker-tier executor):

  • The bare arm is bare of THIS rule, not of the world. It runs in the executor's real baseline — standing global rules, always-loaded caches — since that's the live counterfactual for the fold decision. A bare ✓ therefore means redundant in that environment, not redundant everywhere: one probe's fact-anchoring candidate failed to discriminate purely because the consumer's global rules carry a standing verify-before-relaying order (the general form of the candidate). Record the baseline next to the verdict; never export a non-discrimination result to an environment with a different baseline.
  • Read each arm's stated reasoning before writing the fold, not just its pass/fail. A failing bare arm hands you the exact rationalization the folded rule must refute — one bare arm defended shipping an unverified capability doc with "the document didn't lie — it described the intended behavior," which became the folded rule's named rebuttal. A passing ruled arm can still over-fire (satisfying the rule by refusing the surrounding task outright) — one ruled arm refused to produce any plan at all, which is a wording warning, not a win; the fold was reworded so compliance composes with the task.

Both clauses ship unprobed per the covenant; a Provenance paragraph documents the source incidents and queues their own probes.

Test plan

  • git diff --word-diff against the branch point — zero words removed from the existing bullet, only inserted
  • Line-width and non-ASCII sweep — no new stray characters
  • Both clauses extend the existing bare-probe bullet (matching how prior r4–r6 revisions landed) rather than adding sibling bullets
  • Provenance paragraph added in house style, anonymized consumer framing

🤖 Generated with Claude Code

firaen22 and others added 2 commits July 27, 2026 23:58
…s' reasoning, not just verdicts

Two clauses folded into the existing bare-probe bullet, distilled from a
downstream consumer's first four fold probes of this repo's merged rules:

- The bare arm runs in the executor's REAL baseline (standing global
  rules, always-loaded caches), so a bare-pass verdict means redundant
  in that environment only — record the baseline next to the verdict
  and never export non-discrimination to a different baseline.
- Read each arm's stated reasoning before writing the fold: a failing
  bare arm supplies the exact excuse the rule must refute (carry it in
  near-verbatim); a passing ruled arm that over-fires by refusing the
  surrounding task is a wording warning, not a win.

Provenance paragraph appended; both clauses ship unprobed per the
covenant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third clause on the same bare-probe bullet, and it must ship with the
first one: clause 1 says the bare arm KEEPS its environment (standing
global rules, always-loaded caches); this says strip episodic recall of
the finding under test. Stated apart, each reads as contradicting the
other — the boundary is the rule.

The test is what the arm would have without you: what every invocation
in that environment ships with is baseline; an auto-memory store, a
session summary, or notes from the run being probed are leakage. That
store survives a fresh invocation, so the existing no-shared-state
requirement does not catch it — the arm loads the conclusion, restates
it, and scores as an unaided reproduction.

Countermeasures: recall-dodging framing, an explicit disregard
instruction, and a citation check on the output before the run counts.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@firaen22

Copy link
Copy Markdown
Contributor Author

Pushed a third clause to this branch (16e6e1f) rather than opening a follow-up, because it cannot ship separately from clause 1.

Separate baseline from leakage in the bare arm. Clause 1 says the bare arm keeps its environment (standing global rules, always-loaded caches). This clause says strip episodic recall of the finding under test. Stated apart, each reads as contradicting the other — the boundary between them is the actual rule, so they need to land together.

The test is what the arm would have without you: what every invocation in that environment ships with is baseline; an auto-memory store, a session summary, or notes from the run being probed are leakage. That store survives a "fresh invocation", so the bullet's existing no-shared-state requirement does not catch it — the arm loads the conclusion, restates it, and scores as an unaided reproduction.

Countermeasures added: recall-dodging framing (generic scenario, none of the finding's names or phrasing), an explicit disregard-prior-findings instruction, and a citation check on the output before the run counts — an arm citing the finding or its vocabulary is re-run, not scored.

Provenance updated to cover all three clauses; ships unprobed per the covenant.

A fold probe run at the probing session's own strong tier returned bare
3/3 on a citation-re-resolve rule (redundant with model capability); a
same-day replication of the same fixture at a weak tier returned bare
0/3. Extends the environment-relative-baseline clause: tier is another
axis the verdict does not automatically generalize across, and only a
bare-pass (non-discrimination) result is poisoned by running above the
file's audience — a strong-arm ruled FAIL or bare FAIL still stands.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant