Skip to content

feat: policy replay — measure a change before you ship it - #10

Merged
nayfly merged 1 commit into
mainfrom
fix/lru-eviction-and-effective-fingerprint
Aug 1, 2026
Merged

nayfly merged 1 commit into
mainfrom
fix/lru-eviction-and-effective-fingerprint

Conversation

@nayfly

@nayfly nayfly commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Every policy change this week was validated by hand: write a throwaway script, replay the trail, count what moved. That is the only question that matters — what does this change about decisions I have already seen? — and it caught something real each time: 51 hard_deny that were vocabulary rather than danger, two escalations from parameter names nobody could have guessed.

aetherya policy replay --candidate x.yaml --audit-path audit/decisions.jsonl

Two separations the report insists on:

  • Loosened is not tightened. A decision that gets stricter is friction; one that gets looser is where a regression hides. Counted apart, and the exit code only reacts to adversarial regressions — gating on tightening would make every improvement fail the check meant to enable improvements.
  • Traffic is not the corpus. Traffic measures friction, the corpus measures protection. A change that improves one while quietly breaking the other is the whole reason to replay.

The corpus is evaluated through the free-text path, not as structured calls. The first version got this wrong and reported nine regressions against an identical policy: several cases are only refused because of the parser and intent escalation, and ls / | xargs rm -rf carries no declared tool. A checker that fires on the policy it ships with is worse than no checker.

Replay reconstructs the shape of a recorded call, not the call: the trail stores parameter names and never their values, deliberately, so a credential cannot end up archived. raw_input still carries the command text the content rules read.

Verified in all three directions against the live trail — an identical candidate moves nothing, dropping read from the alias map shows two decisions tightening to hard_deny, and commenting out the rm -rf / family is caught as an adversarial regression with exit 1.

The test button on the breaker. It does not say the policy is right; it says what changed, and it works the same whether a human wrote the change or something proposed it.

Every policy change this week was validated by hand: write a throwaway script,
replay the trail, count what moved. That is the only question that matters —
what does this change about decisions I have already seen? — and it caught
something real each time: 51 hard_deny that were vocabulary rather than danger,
two escalations from parameter names nobody could have guessed.

    aetherya policy replay --candidate x.yaml --audit-path audit/decisions.jsonl

Two separations the report insists on:

- Loosened is not tightened. A decision that gets stricter is friction; one that
  gets looser is where a regression hides. Counted apart, and the exit code only
  reacts to adversarial regressions — gating on tightening would make every
  improvement fail the check meant to enable improvements.
- Traffic is not the corpus. Traffic measures friction, the corpus measures
  protection. A change that improves one while quietly breaking the other is the
  whole reason to replay.

The corpus is evaluated through the free-text path, not as structured calls. The
first version got this wrong and reported nine regressions against an identical
policy: several cases are only refused because of the parser and intent
escalation, and `ls / | xargs rm -rf` carries no declared tool. A checker that
fires on the policy it ships with is worse than no checker.

Replay reconstructs the shape of a recorded call, not the call: the trail stores
parameter names and never their values, deliberately, so a credential cannot end
up archived. `raw_input` still carries the command text the content rules read.

Verified in all three directions against the live trail — an identical candidate
moves nothing, dropping `read` from the alias map shows two decisions tightening
to hard_deny, and commenting out the `rm -rf /` family is caught as an
adversarial regression with exit 1.

The test button on the breaker. It does not say the policy is right; it says
what changed, and it works the same whether a human wrote the change or
something proposed it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@nayfly
nayfly merged commit 9e60363 into main Aug 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant