Fix/lru eviction and effective fingerprint

CHANGELOG.md

@@ -4,6 +4,34 @@ All notable changes to this project are documented in this file.
 
 ## Unreleased
 
+### Added (phase-1 sidecar)
+
+- `enforcement.py` + `enforcement.phase` in policy — the rollout posture is now configuration. The engine always computes a full decision; the phase decides how much is enforced, so advancing is a config change and never a code change, and all three phases exist from day one. `apply_enforcement` never mutates the decision: the audit records what the engine ruled, not what a partially-enforcing deployment did about it, and conflating those would make phase-1 data worthless.
+- `client.py` — `AetheryaClient`, a dependency-free HTTP client for the sidecar shape (agent in one process, engine in another). Fails **closed** from phase 2 on: an unreachable service refuses the action, because a boundary that disappears when the network hiccups is not a boundary. Phase 1 is the explicit exception — it enforces nothing, so an outage must not stop the agent.
+- `rollout_report.py` + `aetherya rollout report` — measures a phase against its exit criteria from the audit trail: window size, decisions by state, what the next phase would act on, distinct actors and tools per state, intent escalations, skipped semantics, policy-fingerprint uniformity, chain integrity, and the `hard_deny` events that require manual review. The `hard_deny_reviewed` criterion **never auto-passes**: the tool can count and list those events, it cannot judge them, and auto-passing would turn the phase gate into the formality it exists to prevent. Exits non-zero until ready, so it works as a promotion gate.
+- `examples/sidecar_agent.py` — the same agent trajectory as `agent_loop.py`, but over HTTP against the container, with the shadow gap counted per run.
+- `Dockerfile.slim` + `config/policy.slim.yaml` — **8.7GB → 330MB**. The size was never ÆTHERYA; it was PyTorch, pulled in by sentence-transformers for the advisory layer. That layer is capped below every deny threshold, so it can escalate to a human but never refuse alone, and the deterministic core does not use it. Verified: the slim image hard-denies exactly what the full one does, including the quote-split evasion `r''m -rf /`, and reports healthy in 8s instead of ~25s. A test pins the two policies to differ in that one field.
+
+### Fixed
+
+- `rollout report` crashed on a malformed audit line or an empty file, because `verify_audit_file` raises rather than returning. Both are findings an operator must see before advancing a phase — they are now reported as a failed `chain_intact` criterion with the detail, instead of taking the tool down or being silently skipped.
+
+### Fixed (CI)
+
+- `examples/agent_loop.py`: the OpenAI backend checked for the SDK before checking for the API key, so the error you got depended on whether `openai` happened to be installed. It is present in a dev environment and absent from CI's `[dev]` extra, which is exactly why `test_openai_backend_requires_a_key` passed locally and failed CI on the last two pushes. The key is now checked first — a missing key is the more common misconfiguration, and you should not need the SDK installed to be told about it — and both paths have tests, one of which blocks the import to prove it.
+- `security_gate` phase 2 generates its own corpus but `AuditLogger` appends by design, so a file left by a previous local run was appended to and counted. The phase then failed on events it had not created, reporting `invalid == total`, which reads like a real integrity regression. CI never saw it (fresh checkout) but a second local run always did. The phase now starts from a clean file and is idempotent, with a test asserting two consecutive runs both pass.
+
+### Fixed (found while containerising)
+
+- **The HTTP API never applied the configured rate limit.** `rate_limit` was validated at load and `build_rate_limiter()` had 13 integration tests, but nothing in the API path ever constructed a limiter — the configured limit was documentation, not behaviour. `AetheryaAPI` now builds one per process (rebuilt only when the policy's rate-limit section changes; rebuilding per request would reset every window and make it a no-op) and passes it to the pipeline. Verified end to end through the container: 65 requests from one actor against a limit of 60 yielded 60 allowed and 5 refused, with the window visible in Redis.
+- `run_pipeline` typed `rate_limiter` as `ActorRateLimiter`, so the Redis backend could not be passed to it at all. Widened to the `RateLimiter` protocol — mypy caught this the moment the API tried to wire it up.
+
+### Added (containerised local stack)
+
+- `Dockerfile` — two-stage build, non-root user, the semantic model baked in at build time (with `require_warm_semantic_model`, downloading it on first boot would either delay readiness or leave the advisory layer silently inactive). Healthcheck asserts `ok && !degraded && policy_fingerprint_match`, so a replica serving under an unintended policy or with the advisory layer inert is not marked healthy.
+- `docker-compose.yml` — decision service plus Redis, with the audit chain on a named volume so it survives the container. Verified: chain intact across a restart (9 events, 0 invalid). The approvals service is deliberately absent — it is localhost-only by design and belongs to phase 3, not phase 1.
+- `config/policy.docker.yaml` — deployment policy, identical to `config/policy.yaml` except for the distributed rate-limit backend. The repo default stays `memory` because `redis` fails closed and would refuse every request without a reachable server. A test pins the two files to differ in exactly that one field so they cannot drift.
+
 ### Added (rollout)
 
 - `docs/rollout-phases.md` — the three-phase production rollout as an executable plan: shadow, hard-deny enforcement, full enforcement. Each phase states its posture, the wiring change it needs, what to measure, and **exit criteria that can fail**, including what to do when shadow mode reports an unacceptable false-positive rate. Also documents the target operating configuration and the `/health` readiness gate.

Dockerfile

@@ -0,0 +1,65 @@
+# ÆTHERYA decision service.
+#
+# Two stages so the runtime image does not carry build tooling. The semantic
+# model is baked in at build time: `require_warm_semantic_model` means the
+# advisory layer declines to run on a cold model, and downloading ~90MB on first
+# boot would either delay readiness or leave the layer silently inactive.
+
+FROM python:3.11-slim AS builder
+
+WORKDIR /build
+
+RUN python -m venv /opt/venv
+ENV PATH="/opt/venv/bin:$PATH"
+
+COPY pyproject.toml README.md ./
+COPY src/ ./src/
+
+RUN pip install --no-cache-dir -U pip \
+    && pip install --no-cache-dir ".[redis]"
+
+# Pre-download the sentence-transformers model into the image.
+ENV HF_HOME=/opt/models
+RUN python -c "from aetherya.constitution import warmup_semantic_model; warmup_semantic_model()"
+
+
+FROM python:3.11-slim AS runtime
+
+# Never run the decision boundary as root.
+RUN useradd --create-home --uid 10001 aetherya
+
+COPY --from=builder /opt/venv /opt/venv
+COPY --from=builder --chown=aetherya:aetherya /opt/models /opt/models
+
+ENV PATH="/opt/venv/bin:$PATH" \
+    PYTHONUNBUFFERED=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    HF_HOME=/opt/models
+
+WORKDIR /app
+COPY --chown=aetherya:aetherya config/ ./config/
+COPY --chown=aetherya:aetherya examples/ ./examples/
+
+# The audit trail is a chained JSONL file; it must outlive the container.
+# docker-compose mounts a volume here.
+RUN mkdir -p /app/audit && chown aetherya:aetherya /app/audit
+VOLUME ["/app/audit"]
+
+USER aetherya
+EXPOSE 8080
+
+# `degraded` and `policy_fingerprint_match` are the fields that matter: a replica
+# serving with the advisory layer inactive, or under an unintended policy, should
+# not receive traffic.
+HEALTHCHECK --interval=15s --timeout=5s --start-period=20s --retries=3 \
+    CMD python -c "import json,urllib.request;\
+d=json.load(urllib.request.urlopen('http://127.0.0.1:8080/health',timeout=4));\
+raise SystemExit(0 if d.get('ok') and not d.get('degraded') and d.get('policy_fingerprint_match') else 1)"
+
+# --require-semantic-ready: fail startup rather than serve in a degraded state.
+CMD ["aetherya-api", \
+     "--host", "0.0.0.0", \
+     "--port", "8080", \
+     "--policy-path", "config/policy.yaml", \
+     "--audit-path", "audit/decisions.jsonl", \
+     "--require-semantic-ready"]

Dockerfile.slim

@@ -0,0 +1,70 @@
+# ÆTHERYA decision service — slim variant (~200MB instead of ~8.7GB).
+#
+# The size of the full image is not ÆTHERYA: it is PyTorch, pulled in by
+# sentence-transformers for the semantic layer. That layer is advisory by
+# design — capped below every deny threshold, so it can escalate to a human
+# but never refuse on its own. The deterministic core (parser, guards,
+# procedural corpus, jailbreak, capabilities, risk, confirmation, audit) does
+# not use it at all.
+#
+# So this variant drops the dependency and runs with `use_semantic: false`.
+# Everything that hard-denies today still hard-denies: those verdicts come
+# from the procedural and jailbreak corpora, not from similarity. What you
+# lose is paraphrase detection on inputs no keyword matches.
+#
+# Recommended for phase 1: what you are measuring there is deterministic
+# refusals, and this starts in seconds instead of ~15s.
+
+FROM python:3.11-slim AS builder
+
+WORKDIR /build
+
+RUN python -m venv /opt/venv
+ENV PATH="/opt/venv/bin:$PATH"
+
+COPY pyproject.toml README.md ./
+COPY src/ ./src/
+
+RUN pip install --no-cache-dir -U pip \
+    && pip install --no-cache-dir --no-deps ".[redis]" \
+    && pip install --no-cache-dir "pyyaml" "numpy>=1.26.0" "redis>=5.0.0"
+
+
+FROM python:3.11-slim AS runtime
+
+# Never run the decision boundary as root.
+RUN useradd --create-home --uid 10001 aetherya
+
+COPY --from=builder /opt/venv /opt/venv
+
+ENV PATH="/opt/venv/bin:$PATH" \
+    PYTHONUNBUFFERED=1 \
+    PYTHONDONTWRITEBYTECODE=1
+
+WORKDIR /app
+COPY --chown=aetherya:aetherya config/ ./config/
+COPY --chown=aetherya:aetherya examples/ ./examples/
+
+# The audit trail is a chained JSONL file; it must outlive the container.
+# docker-compose mounts a volume here.
+RUN mkdir -p /app/audit && chown aetherya:aetherya /app/audit
+VOLUME ["/app/audit"]
+
+USER aetherya
+EXPOSE 8080
+
+# `degraded` and `policy_fingerprint_match` are the fields that matter: a replica
+# serving with the advisory layer inactive, or under an unintended policy, should
+# not receive traffic.
+HEALTHCHECK --interval=15s --timeout=5s --start-period=20s --retries=3 \
+    CMD python -c "import json,urllib.request;\
+d=json.load(urllib.request.urlopen('http://127.0.0.1:8080/health',timeout=4));\
+raise SystemExit(0 if d.get('ok') and not d.get('degraded') and d.get('policy_fingerprint_match') else 1)"
+
+# --require-semantic-ready: fail startup rather than serve in a degraded state.
+CMD ["aetherya-api", \
+     "--host", "0.0.0.0", \
+     "--port", "8080", \
+     "--policy-path", "config/policy.slim.yaml", \
+     "--audit-path", "audit/decisions.jsonl", \
+     "--no-warmup-semantic"]

config/policy.docker.yaml

@@ -0,0 +1,219 @@
+# Deployment policy: identical to config/policy.yaml except that the rate
+# limiter uses the distributed backend. Mounted by docker-compose.yml.
+#
+# The repo default stays `memory` so the CLI and a bare `aetherya-api` work
+# with no infrastructure. `redis` fails closed, so pointing at it without a
+# reachable server refuses every request — correct, but not a default.
+#
+# tests/test_deployment_contracts.py asserts these two files differ in
+# exactly this one field, so they cannot drift apart.
+version: 1
+
+modes:
+  consultive:
+    default_state: log_only
+    thresholds:
+      deny_at: 90
+      confirm_at: 60
+      log_only_at: 0
+
+  operative:
+    default_state: allow
+    thresholds:
+      deny_at: 80
+      confirm_at: 50
+      log_only_at: 0
+
+aggregator:
+  weights:
+    execution_gate: 1.0
+    capability_gate: 1.0
+    jailbreak_guard: 1.0
+    procedural_guard: 1.0
+    constitution: 1.0
+  hard_deny_if:
+    - critical_tag_detected
+    - "tag:jailbreak_attempt"
+    - "tag:tool_not_allowed"
+    - "tag:capability_tool_denied"
+
+procedural_guard:
+  # Each entry activates a rule family in procedural_guard.py. The legacy
+  # literals ("rm -rf /", "mkfs", ":(){:|:&};:") now expand to flag-order
+  # tolerant, device-anchored patterns instead of naive substring matching.
+  # Symbolic keys activate families that have no single literal form. An
+  # unrecognised entry degrades to literal substring matching, so custom
+  # operator rules keep working unchanged.
+  critical_tags:
+    - "rm -rf /"                # root recursive delete: -rf, -fr, -r -f, --no-preserve-root
+    - "mkfs"                    # filesystem format — requires a /dev/ argument
+    - ":(){:|:&};:"             # classic fork bomb
+    - "block_device_write"      # dd of=/dev/sdX, > /dev/sdX, shred|wipefs|tee|truncate /dev/X
+    - "interpreter_destruction" # shutil.rmtree('/') inside a python -c one-liner
+    - "find_delete"             # find / ... -delete
+    - "recursive_force_delete"  # rm -rf on a non-root path (risk 55, not hard-deny)
+    - "remote_code_execution"   # curl|wget piped into a shell (risk 70)
+  privileged_ops:
+    - "apt remove"
+    - "apt purge"
+    - "userdel"
+    - "chmod -R 777 /"
+
+execution_gate:
+  enabled: true
+  allowed_tools:
+    - shell
+    - http
+    - filesystem
+  require_target_for_operate: false
+  required_parameters:
+    shell:
+      - command
+  allowed_parameters:
+    shell:
+      - command
+      - timeout
+    http:
+      - method
+      - url
+      - body
+    filesystem:
+      - path
+      - operation
+      - confirm_token
+      - confirm_context
+
+capability_matrix:
+  enabled: true
+  default_allow: false
+  roles:
+    operator:
+      tools:
+        - shell
+        - http
+        - filesystem
+      operations:
+        - read
+        - write
+        - append
+        - list
+        - request
+        - command
+  actors:
+    robert:
+      roles:
+        - operator
+      tools: []
+      operations: []
+
+confirmation:
+  enabled: true
+  on_confirmed: allow
+  require_for:
+    decisions:
+      - require_confirm
+    tools: []
+    operations:
+      - delete
+      - write
+    min_risk_score: 0
+  evidence:
+    token_param: confirm_token
+    context_param: confirm_context
+    token_pattern: "^ack:[a-z0-9_-]{8,}$"
+    min_context_length: 12
+    signed_proof:
+      enabled: false
+      proof_param: confirm_proof
+      key_env: AETHERYA_CONFIRMATION_HMAC_KEY
+      keyring_env: AETHERYA_CONFIRMATION_HMAC_KEYRING
+      active_kid: k1
+      max_valid_for_sec: 900
+      clock_skew_sec: 5
+      replay_mode: single_use
+      replay_store: memory
+      replay_redis_url_env: AETHERYA_CONFIRMATION_REPLAY_REDIS_URL
+      replay_redis_prefix: aetherya:appr
+
+# LLM shadow — non-authoritative ethical evaluation, never affects the decision.
+# provider: dry_run | openai | anthropic
+#   openai    requires OPENAI_API_KEY    (e.g. model: gpt-4o-mini)
+#   anthropic requires ANTHROPIC_API_KEY (e.g. model: claude-opus-4-8) and
+#             `pip install -e ".[anthropic]"`. Note: temperature is ignored by
+#             the anthropic provider (current Claude models reject sampling params).
+llm_shadow:
+  enabled: false
+  provider: dry_run
+  model: gpt-dry
+  temperature: 0.0
+  max_tokens: 96
+  timeout_sec: 10.0
+
+policy_adapter_shadow:
+  enabled: false
+  max_signals: 3
+
+# Rate limiting. The "memory" backend keeps windows in process memory, so behind
+# N workers the effective limit is N x requests_per_window — the configured limit
+# is simply not the limit that applies. Any multi-worker or multi-replica
+# deployment must use "redis". The redis backend fails closed: if Redis is
+# unreachable the request is refused rather than silently going unlimited.
+rate_limit:
+  backend: redis               # memory | redis
+  requests_per_window: 60
+  window_seconds: 60.0
+  max_actors: 10000            # memory backend only: LRU cap against actor-id flooding
+  redis_url_env: AETHERYA_RATE_LIMIT_REDIS_URL
+  redis_prefix: "aetherya:ratelimit"
+
+# Rollout posture. The engine always computes a full decision; the phase only
+# decides how much of it is enforced, so advancing is a config change and never
+# a code change. Measure the current phase with `aetherya rollout report` before
+# advancing. See docs/rollout-phases.md.
+#   1 shadow    — observe only, nothing is refused
+#   2 hard_deny — refuse hard_deny
+#   3 full      — refuse hard_deny and deny, hold escalate for confirmation
+enforcement:
+  phase: 1
+
+# Intent escalation — decouples the guard chain from the parser's verb list.
+# ExecutionGate and the capability matrix only evaluate `operate` requests, so
+# without this stage a command the parser does not recognise (e.g.
+# `dd if=/dev/zero of=/dev/sda`) skips both gates and can reach `allow`.
+# Escalation is monotone: it only ever raises ask/consultive -> operate/operative.
+intent_escalation:
+  enabled: true
+  use_procedural_signal: true   # any ProceduralGuard hit implies operative content
+  use_shape_signals: true       # $(...), pipe-to-shell, > /path, binary+flags, sudo X
+
+# OutputGate contract configuration.
+# require_candidate_response: false (default) means OutputGate is opt-in —
+# the caller chooses whether to pass candidate_response to run_pipeline.
+# Set to true in deployments where output protection must always be enforced.
+output_gate:
+  require_candidate_response: false
+
+# Constitution evaluator configuration.
+# use_semantic: activate the SemanticEvaluator (all-MiniLM-L6-v2) for inputs that
+#   Fast keyword matching cannot resolve definitively. Enabled by default.
+# Thresholds govern the SemanticEvaluator similarity cutoffs.
+# Modify with care: lowering thresholds increases sensitivity and false-positive risk.
+constitution:
+  use_semantic: true
+  semantic_violation_threshold: 0.55
+  semantic_gray_zone_threshold: 0.35
+  # NON-AUTHORITATIVE CONTRACT: the semantic layer is the only learned component
+  # in the decision path. Its risk contribution is capped here so it can raise a
+  # request to `escalate` (a human decides) but never to `deny` on its own.
+  # Must stay strictly below every mode's deny_at — validated at policy load.
+  # This is what keeps the *core verdict* deterministic: swapping the model can
+  # change whether a human is asked, never whether an action is refused.
+  semantic_max_risk: 60
+  # Loading all-MiniLM-L6-v2 costs ~5s in a fresh process. That belongs at
+  # deployment startup (`aetherya warmup`), never inside a decision. With this
+  # set, the advisory layer declines to run when the model is not already in the
+  # process cache instead of stalling the decision path; the trace then carries
+  # `constitution.semantic_skipped: model_not_warm`. Long-lived servers should
+  # call `aetherya warmup` at boot so the layer is always available.
+  # Set to false only if you accept a multi-second cold load inside a decision.
+  require_warm_semantic_model: true