Fix/guard decoupling and detection corpus

CHANGELOG.md

@@ -4,8 +4,58 @@ All notable changes to this project are documented in this file.
 
 ## Unreleased
 
+### Deployment readiness
+
+- **Distributed rate limiting.** `RedisActorRateLimiter` shares the sliding window across processes via a sorted set per actor, with the drop/count/add/expire sequence in one pipeline so concurrent workers cannot read a stale count. The in-process limiter meant that behind `uvicorn --workers N` the effective limit was `N x requests_per_window` — the configured limit simply was not the limit that applied, and it scaled with replica count. Select via `rate_limit.backend: redis` (`AETHERYA_RATE_LIMIT_REDIS_URL`). **Fails closed**: an unreachable Redis refuses the request rather than silently going unlimited. A throttled request withdraws its own entry so a caller hammering the endpoint cannot extend its own window.
+- **Policy version contract.** `AETHERYA_EXPECTED_POLICY_FINGERPRINT` (or `load_policy_config(..., expected_fingerprint=...)`) pins the policy a process may run; a mismatch raises `PolicyFingerprintMismatch` at load. Two replicas on different policies return different decisions for identical input and nothing surfaces that until someone diffs audit trails — pinning turns silent divergence into a startup failure. `/health` reports `policy_fingerprint_pinned` and `policy_fingerprint_match`.
+- **Semantic layer readiness is now observable.** With `require_warm_semantic_model` the advisory layer declines to run on a cold model, which made it a silent no-op in any deployment that never warmed it. `/health` now reports `semantic_enabled`, `semantic_ready`, `semantic_model_warm` and `degraded`. The API server preloads the model before accepting traffic (`--no-warmup-semantic` to skip); `--require-semantic-ready` fails startup instead of serving degraded.
+- **Audit durability and retention.** `AuditLogger` accepts `fsync=True` (off by default) and `mirrors=[...]`, an `AuditSink` fan-out for shipping each event to append-only storage (S3 object-lock, WORM log service). Mirrors receive the byte-identical line the primary wrote, so the chain verifies the same on either copy. Mirror failures increment `mirror_errors` rather than raising — a transient archive outage must not take the decision path down, but the counter must be monitored. The chain tip only advances after the authoritative write succeeds.
+
+### Security
+
+- **Adversarial sweep of the procedural corpus.** 26 systematic evasions were run against the new rules; 4 got through and are now closed, and 5 more were downgraded from `hard_deny` to `escalate` by quoting. Fixed: `r''m -rf /` (quote-split command name, previously **allowed outright**), `truncate -s 0 /dev/sda`, `echo x | tee /dev/sda`, and `python -c "shutil.rmtree('/')"`. The corpus is pinned as a regression fixture (`tests/fixtures/security_corpus/v1/procedural_evasions.json`) with per-case expectations; cases whose destructive intent is only resolvable at execution time (variable expansion, pipe contents, base64) are asserted to reach the confirmation band rather than hard-deny.
+- **Quote and escape stripping moved into the shared normalizer.** Quoting is syntactic to a shell and inert in prose, but it splits any literal a pattern is anchored on. The same evasion worked against the **JailbreakGuard** (`ign''ore previous instructions` was undetected); both guards now normalize identically, which is the property `text_normalize` exists to guarantee. New procedural rules: `block_device_write_tool` (tee/truncate to a device), `interpreter_destruction` (`rmtree('/')`), `find_delete` (`find / -delete`).
+- `Parser`: parameter and target values are no longer lowercased. Values were extracted from a lowercased copy of the input, so `param.path=/tmp/MyFile.TXT` was recorded and audited as `/tmp/myfile.txt` — a different file on any case-sensitive filesystem, meaning the audit trail did not describe the action that was authorised. Field *names* remain case-insensitive and tool names are still lowercased to match the execution allowlist.
+
 ### Added
 
+- **Structured input as the preferred integration path.** `run_pipeline_structured(action, ...)` and `run_pipeline(..., action=...)` take a fully-formed `ActionRequest`, skipping the heuristic parser entirely so no part of the decision depends on inferring structure from free text. The HTTP API accepts an `action` object on `POST /v1/decide` alongside `raw_input`; the response reports `meta.input_mode`. Every other stage is unchanged: structured callers still pass the ABI contract and IntentEscalation, so declaring `intent="ask"` on a destructive payload bypasses nothing.
+- `config.py`: `RateLimitBackendConfig` (loaded from `rate_limit`), `POLICY_FINGERPRINT_ENV`, `PolicyFingerprintMismatch`, `expected_policy_fingerprint()`.
+- `rate_limiter.py`: `RateLimiter` protocol, `RedisActorRateLimiter`, `build_rate_limiter()`.
+- `audit.py`: `AuditSink` protocol, `AuditLogger.fsync`, `AuditLogger.mirrors`, `AuditLogger.mirror_errors`.
+- `api_server.py`: `warmup_semantic_layer()`, `--no-warmup-semantic`, `--require-semantic-ready`.
+
+### Security (guard chain)
+
+- **`ProceduralGuard` rewritten from substring matching to an anchored rule corpus.** The previous implementation tested `critical_tag.lower() in text`, which both missed real commands and fired on prose. Verified gaps that are now closed: `rm -fr /`, `rm -r -f /`, `rm --recursive --force /`, `rm -rf --no-preserve-root /` and extra-whitespace variants all degraded from `hard_deny` to `escalate`; `dd if=/dev/zero of=/dev/sda` was **allowed** outright, as were `> /dev/sda`, `shred`/`wipefs` on a device, and `mkfs.ext4 /dev/sda1`. Root-recursive-delete detection now parses `rm` invocations (flag clustering in any order, long flags, `--no-preserve-root`) and classifies the target against a set of root-like system paths. Matching runs over the shared security normalizer, so invisible-character obfuscation cannot split a command token.
+- `ProceduralGuard`: severity tiering. `critical_tag_detected` is a hard-deny tag and is now reserved for irreversible destruction. `rm -rf` on a non-root path (`recursive_force_delete`, risk 55) and `curl|wget` piped into a shell (`remote_code_execution`, risk 70) land in the confirmation band instead of being hard-denied or ignored.
+- `ProceduralGuard`: false positives removed. `explain what mkfs does` previously scored 100 via bare substring match; `mkfs` now requires a `/dev/` argument and `rm` rules require a path-like target, so `What does rm -rf do?` no longer fires.
+- `ProceduralGuard`: all matching rules are collected. Score, confidence and reason come from the highest-severity match; the tag list is the de-duplicated union across every match, mirroring the JailbreakGuard contract.
+- **New `intent_escalation` pipeline stage — the guard chain no longer depends on the parser's verb list.** `ExecutionGate` and `CapabilityGate` only evaluate `intent=operate`, so an executable command using a verb the parser did not know (`dd if=/dev/zero of=/dev/sda`) was classified `ask`, skipped both gates and reached `allow`. The stage re-derives operative intent from raw input using command shape — command substitution, pipes into a shell, redirects to absolute paths, known binaries carrying flags, device operands, `sudo` prefixes — plus any `ProceduralGuard` hit. The transform is monotone (`ask` → `operate` only), so it cannot be used to relax a request. Escalations are recorded in the audit context under `intent_escalation`.
+- **`ActorRateLimiter` state is now bounded.** Windows were held in a plain dict keyed by actor and never released, so rotating the actor field grew the map without limit (measured: 50 000 distinct actors retained 50 000 deques indefinitely). Windows now live in an LRU map capped at `max_actors` (default 10 000) with fully-expired entries swept every `sweep_every` checks (default 1 000). LRU order is a security property, not just a memory one: an actor being actively rate-limited is by definition recently used, so flooding cannot evict — and thereby reset — the window of the actor it is trying to displace.
+
+### Fixed
+
+- `Parser`: meta-questions about a command no longer escalate. v0.8.0 classified any mention of an operative verb as operative, so `"What does rm -rf do?"` became an operative request with no declared tool — which scores 55 at the ExecutionGate, above the operative `confirm_at` of 50, and therefore always returned `escalate`. A narrow frame (`what does/is/are …`, `how does … work`, `explain/describe/define …`, anchored at the start, third person only) now keeps such inputs consultive. It is disqualified by any clause separator (`;`, `&&`, `|`, newline, `and then`, `then`), by an explicit `tool:` or `mode:operative`, and does not match first-person how-to phrasing, so `"How do I run a Docker container?"` and `"Can you delete all logs?"` are unchanged. This is an ergonomics fix, not a security boundary: `ProceduralGuard` matches the raw input regardless of intent and `IntentEscalation` raises the request back to `operate`, so `"explain rm -rf /"` is still hard-denied.
+- `scripts/pre_api_gate.py`: the `shadow_timeout` scenario compared absolute subprocess wall time against fixed budgets, which measured interpreter startup and package imports rather than what the scenario asserts — that the shadow adds bounded latency and that `--no-wait-shadow` adds none. On a host where importing `aetherya.cli` alone costs 130 ms, the 150 ms `--no-wait-shadow` budget could not be met regardless of engine behaviour. Budgets now apply to the overhead over a shadow-disabled baseline invocation, and the wait path additionally excludes the measured provider-SDK import cost. The gate is hardware-independent as a result: measured overhead is 26 ms (no-wait) and 291 ms (wait, against a configured 100 ms shadow timeout), versus 157 ms and 733 ms of absolute wall time.
+
+### Changed
+
+- **The semantic constitution layer is now non-authoritative.** It is the only component in the decision path whose output depends on a learned model, so it can no longer decide alone: its risk contribution is capped at the new `constitution.semantic_max_risk` (default 60) and it emits only the non-hard-deny tag `semantic_advisory`. The cap is validated at policy load to sit strictly below every mode's `deny_at` — a policy that would grant the model authority to deny is rejected rather than silently accepted. Net effect: swapping the model can change whether a human is asked, never whether an action is refused outright.
+- **The semantic layer never blocks a decision on a cold model load.** Loading `all-MiniLM-L6-v2` costs ~5 s in a fresh process; with `use_semantic` on by default this landed inside the decision path and made one-shot CLI use unusable (measured 5.6 s per `aetherya decide`, versus 0.15 s with the layer off). New `constitution.require_warm_semantic_model` (default `true`) makes the layer decline to run when the model is not already in the process cache, recording `constitution.semantic_skipped: model_not_warm` in the trace. Long-lived deployments call `aetherya warmup` at boot to make it available. CLI cold-start returns to ~0.16 s.
+- **`FastKeywordEvaluator` ambiguity is decided by evidence, not text length.** Ambiguity was gated on `token_count < 10`, so inputs longer than ten tokens never reached the semantic layer — precisely where a paraphrased attack has the most room to hide. An input with no keyword evidence either way is now ambiguous regardless of length; length only modulates the confidence the fast layer reports in its own "clean" verdict. A keyword found only in negated form remains a definitive allow.
+- `config/policy.yaml`: `procedural_guard.critical_tags` entries now activate rule families rather than literal substrings. The three shipped literals are unchanged in meaning; three symbolic families are added (`block_device_write`, `recursive_force_delete`, `remote_code_execution`). Unrecognised entries degrade to literal substring matching, so custom operator rules keep working.
+
+### Added
+
+- `text_normalize.py` — `normalize_security_text()`, extracted from `jailbreak.py` and now shared with `ProceduralGuard`, so both guards match against one canonical form and a bypass fixed in one is fixed in the other.
+- `intent_escalation.py` — `IntentEscalator`, `IntentEscalationOutcome`, `apply_escalation()`.
+- `config.py`: `IntentEscalationConfig` (`enabled`, `use_procedural_signal`, `use_shape_signals`), loaded from `policy.yaml` under `intent_escalation`.
+- `config.py`: `ConstitutionConfig.semantic_max_risk` and `ConstitutionConfig.require_warm_semantic_model`, plus `_validate_semantic_authority()` enforcing the cap against every mode's `deny_at` at load time.
+- `constitution.py`: `DEFAULT_SEMANTIC_MAX_RISK`, `SEMANTIC_ADVISORY_TAG`, `is_model_warm()`, and `SemanticEvaluator.can_evaluate()`.
+- `rate_limiter.py`: `RateLimitConfig.max_actors`, `RateLimitConfig.sweep_every`, and `ActorRateLimiter.tracked_actors()` for observability.
+- Audit trace: `context.intent_escalation` when a request was escalated; `context.constitution.semantic_model` naming the model behind an advisory signal; `context.constitution.semantic_skipped` distinguishing "the advisory layer declined to run" from "it found nothing".
+
 - `audit_divergence.py` — ethical divergence telemetry over the audit trail. New command `aetherya audit divergence` (also `python -m aetherya.audit_divergence`) aggregates LLM-shadow results: shadow evaluation count, shadow errors, state mismatch rate, mean/max `risk_delta`, parse success rate, flag frequency, and top-N events by absolute risk delta (`--top`, `--min-abs-delta`, `--json`).
 - Pipeline: the audited `llm_shadow` block now includes an `evaluation` sub-block (`parse_success`, `reasoning`, `flags`) when the provider performs a structured ethical evaluation. Dry-run shadow events are unchanged.
 - `aetherya warmup` — preloads the semantic constitution model (`all-MiniLM-L6-v2` by default, `--model-name` to override) into the process cache and runs one dummy encode. Intended for deployment startup: with `use_semantic` enabled by default, the first ambiguous input would otherwise trigger the model download/load inside the decision path. Exposed as `warmup_semantic_model()` in `constitution.py`.

README.md

@@ -62,14 +62,25 @@ python -m aetherya.audit_verify --audit-path audit/decisions.jsonl --require-cha
 ## Pipeline (simplified)
 
 ```
-Input → Parser → RateLimiter → ExecutionGate → CapabilityGate
-      → JailbreakGuard → ProceduralGuard → Constitution
+Input → Parser → IntentEscalation → RateLimiter → ExecutionGate
+      → CapabilityGate → JailbreakGuard → ProceduralGuard → Constitution
       → RiskAggregator → ConfirmationGate → PolicyEngine → Decision
       → Explainability → Audit
 ```
 
 Fail-closed: any exception in any stage → `allowed=false`.
 
+**IntentEscalation** decouples the guard chain from the parser: `ExecutionGate`
+and `CapabilityGate` only evaluate `operate` requests, so a command the parser
+did not recognise would otherwise skip them entirely. Escalation is monotone —
+it only ever tightens. See [docs/security-model.md](docs/security-model.md).
+
+**Determinism.** The keyword/guard/policy path is fully deterministic. The
+optional semantic layer is the one learned component, and it is deliberately
+non-authoritative: its risk is capped below every mode's `deny_at`, so it can
+escalate to a human but never deny on its own. It is also never allowed to block
+a decision on a cold model load — run `aetherya warmup` at startup to enable it.
+
 ---
 
 ## Approval Demo (DENY → SIGN → ALLOW → REPLAY DENY → AUDIT OK)

config/policy.yaml

@@ -29,10 +29,21 @@ aggregator:
     - "tag:capability_tool_denied"
 
 procedural_guard:
+  # Each entry activates a rule family in procedural_guard.py. The legacy
+  # literals ("rm -rf /", "mkfs", ":(){:|:&};:") now expand to flag-order
+  # tolerant, device-anchored patterns instead of naive substring matching.
+  # Symbolic keys activate families that have no single literal form. An
+  # unrecognised entry degrades to literal substring matching, so custom
+  # operator rules keep working unchanged.
   critical_tags:
-    - "rm -rf /"
-    - "mkfs"
-    - ":(){:|:&};:"   # fork bomb clásica
+    - "rm -rf /"                # root recursive delete: -rf, -fr, -r -f, --no-preserve-root
+    - "mkfs"                    # filesystem format — requires a /dev/ argument
+    - ":(){:|:&};:"             # classic fork bomb
+    - "block_device_write"      # dd of=/dev/sdX, > /dev/sdX, shred|wipefs|tee|truncate /dev/X
+    - "interpreter_destruction" # shutil.rmtree('/') inside a python -c one-liner
+    - "find_delete"             # find / ... -delete
+    - "recursive_force_delete"  # rm -rf on a non-root path (risk 55, not hard-deny)
+    - "remote_code_execution"   # curl|wget piped into a shell (risk 70)
   privileged_ops:
     - "apt remove"
     - "apt purge"
@@ -133,6 +144,29 @@ policy_adapter_shadow:
   enabled: false
   max_signals: 3
 
+# Rate limiting. The "memory" backend keeps windows in process memory, so behind
+# N workers the effective limit is N x requests_per_window — the configured limit
+# is simply not the limit that applies. Any multi-worker or multi-replica
+# deployment must use "redis". The redis backend fails closed: if Redis is
+# unreachable the request is refused rather than silently going unlimited.
+rate_limit:
+  backend: memory              # memory | redis
+  requests_per_window: 60
+  window_seconds: 60.0
+  max_actors: 10000            # memory backend only: LRU cap against actor-id flooding
+  redis_url_env: AETHERYA_RATE_LIMIT_REDIS_URL
+  redis_prefix: "aetherya:ratelimit"
+
+# Intent escalation — decouples the guard chain from the parser's verb list.
+# ExecutionGate and the capability matrix only evaluate `operate` requests, so
+# without this stage a command the parser does not recognise (e.g.
+# `dd if=/dev/zero of=/dev/sda`) skips both gates and can reach `allow`.
+# Escalation is monotone: it only ever raises ask/consultive -> operate/operative.
+intent_escalation:
+  enabled: true
+  use_procedural_signal: true   # any ProceduralGuard hit implies operative content
+  use_shape_signals: true       # $(...), pipe-to-shell, > /path, binary+flags, sudo X
+
 # OutputGate contract configuration.
 # require_candidate_response: false (default) means OutputGate is opt-in —
 # the caller chooses whether to pass candidate_response to run_pipeline.
@@ -149,3 +183,18 @@ constitution:
   use_semantic: true
   semantic_violation_threshold: 0.55
   semantic_gray_zone_threshold: 0.35
+  # NON-AUTHORITATIVE CONTRACT: the semantic layer is the only learned component
+  # in the decision path. Its risk contribution is capped here so it can raise a
+  # request to `escalate` (a human decides) but never to `deny` on its own.
+  # Must stay strictly below every mode's deny_at — validated at policy load.
+  # This is what keeps the *core verdict* deterministic: swapping the model can
+  # change whether a human is asked, never whether an action is refused.
+  semantic_max_risk: 60
+  # Loading all-MiniLM-L6-v2 costs ~5s in a fresh process. That belongs at
+  # deployment startup (`aetherya warmup`), never inside a decision. With this
+  # set, the advisory layer declines to run when the model is not already in the
+  # process cache instead of stalling the decision path; the trace then carries
+  # `constitution.semantic_skipped: model_not_warm`. Long-lived servers should
+  # call `aetherya warmup` at boot so the layer is always available.
+  # Set to false only if you accept a multi-second cold load inside a decision.
+  require_warm_semantic_model: true