skill-authoring: repair the derived-file rule's trigger (it fails both arms), record first probe results for the 2026-07-25 four - #85
Open
firaen22 wants to merge 1 commit into
Conversation
… probe results
The four rules folded 2026-07-25 shipped `unprobed` per the covenant. This is
that probe, run against the merged text (n=1/arm, one weaker-tier executor —
screens for a large effect only).
- §7 derived-file rule: failed BOTH arms. Given a wrong value in a file with no
provenance cue, each arm edited it in place. The written trigger named the
classification ("a derived file") that the rule exists to force, so it fired
only when provenance was already known. Trigger restated as the observable
state; body unchanged. Repaired form ships unprobed.
- §3 bound-invalidation, §7 bare-executor-probe: discriminated. Markers updated.
- §3 world-fact staleness: did NOT discriminate — a bare arm produced the
scoping unaided. Flagged in its marker for a demotion judgment, not demoted
at n=1.
- §2: derive a cross-reference's section number from the file, never from
recall. One such reference reached main and cost fix commit 8f8413f.
Provenance amended with the results, including two discarded runs (round-1
scenarios leaked the taught distinction; one bare arm was contaminated by the
author's own session memory reaching the subagent).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four rules folded on 2026-07-25 (#82, merged via #84) shipped with in-body
unprobedmarkers. This is that probe, run against the merged text rather thanwhat I submitted — the fold gate changed those rules materially, so the
submitted version was not the right thing to test.
One rule fails. The repair is the substance of this PR; the rest is recording
what the probe found.
§7 derived-file rule — failed both arms, trigger repaired
Round 1 said "a generated config file says the timeout is 30s; the running
system says 5s. Fix it." Both arms passed. That scenario leaks: it hands over
the provenance the rule exists to make you ask about. Discarded and re-run.
Round 2 said
src/config/timeouts.tswith no provenance cue. Both armshand-edited the value. Neither asked what produces the file.
The written trigger was "a derived file contradicting reality" — but
recognizing a file as derived is the discovery the rule exists to force. So
it fires when someone has already told you the file is generated (exactly when
you no longer need it) and stays silent when they haven't (exactly when you
do). Generalizable defect: a trigger naming a property the executor can only
know by having already applied the rule.
The repair restates the trigger as the observable state — content contradicts
reality, correction imminent, therefore establish provenance first — and keeps
the body, which was never the problem. The repaired form ships
unprobed; itsmarker says so.
The other three
The bare-probe rule's own bare arm reached the instinct — "correct and
non-duplicate clears the bar for truth, not for inclusion" — and then settled
the question by judgment instead of running the two arms. Armed, and failed.
World-fact scoping did not discriminate: on a scenario that doesn't pre-split
the files, a bare arm produced the prioritization unaided. At n=1 that is a
flag for a demotion judgment, not grounds to demote, so its marker records the
result and the rule stays. Your call whether it earns §3's line.
§2 — derive cross-references from the file, never from recall
Folded onto §2's existing verify-everything bullet rather than shipped as its
own rule: a pointer inside a rule is a claim about the target file, and the
same verification applies. Incident is in this repo — I wrote "§4's sync
contract" when the clause is in §3, from memory, while the correct line sat
already-quoted in my own working context. Cost you fix commit
8f8413f. Asecond instance the same day I caught in my own draft (contributor-reported;
only the first is verifiable here).
A candidate this probe killed
Across the four, the two that discriminate hand the executor a procedure;
the one that doesn't hands it an observation a careful bare executor states
unaided. That looked like a cheap pre-filter for the probe queue, so I drafted
it as a rule and probed it the same day — and a clean bare arm produced its
substance unaided. Non-discriminating by this section's own verdict table, so
it is recorded in Provenance as an observed pattern and not shipped as a
rule.
Method, including what went wrong
n=1 per arm, one weaker-tier executor, independent fresh invocations. Screens
for a large effect only. Two runs were discarded:
under test.
very rule from its own session memory, verbatim. A subagent bare arm is not
bare by default; it needs framing that dodges recall relevance, an explicit
disregard instruction, and a citation check on the output before the run
counts. Noted in the Provenance entry, since anyone re-running these probes
will hit it.
I also had to override a clause in my own pre-registration mid-grading: I had
frozen "an answer that never establishes provenance is not armed, excluded",
which conflates not armed with failed and would have excluded every failing
bare arm while counting the passes — a change detector inside the gate, written
while building the gate for it. Armed tests whether the scenario presented
the guarded situation, never whether the answer handled it.
Reviewed fresh-context before submitting; three findings fixed, including an
overclaim in my own §2 addition (I had asserted both dangling references as
repo-verifiable when only one is).