Add Evidence–Claim Divergence (ECD) estimator, docs, and tests

docs/EVIDENCE_CLAIM_DIVERGENCE.md

@@ -0,0 +1,100 @@
+# Evidence–Claim Divergence
+
+## A reproducible framework for measuring calibration in autonomous systems
+
+**Status:** T2 research instrument. This document separates the scientific proposal from the reference measurement substrate implemented in `verifiable/ecd.py`.
+
+## Abstract
+
+Reliable evaluation of advanced AI systems requires more than measuring task accuracy or benchmark performance. Existing evaluation methods primarily assess whether a system reaches a correct outcome, while providing limited insight into whether the system's observable claims remain consistent with independently verifiable evidence throughout execution.
+
+Evidence–Claim Divergence (ECD) is an observable phenomenon: structural disagreement between externally observable system claims and independently verifiable execution evidence. Hallucination Distance (HD) is a family of empirical estimators over observable claims and evidence graphs. The Evidence–Reasoning Benchmark (ERB) is the proposed evaluation protocol for deterministic traces, evidence graphs, controlled perturbations, and public datasets. AEGIS-Ω is the reference measurement substrate that reduces measurement variance with deterministic replay, canonical serialization, provenance tracking, and reproducible evidence generation.
+
+## Formal objects
+
+Let `C ∈ 𝒞` represent observable claims emitted by an autonomous system, including textual assertions, API calls, state declarations, externally visible outputs, and structured metadata.
+
+Let `E ∈ ℰ` represent an immutable evidence graph constructed from independently observed execution artifacts, including telemetry, logs, execution traces, provenance records, attestations, and verified state transitions.
+
+ECD is defined as:
+
+```text
+ECD(C, E) = inf over M∈𝓜 of d(M(C), E)
+```
+
+where `M` maps heterogeneous claims into graph space and `d` is a graph distance over attributed evidence graphs. ECD is therefore a latent structural property of the claim/evidence boundary rather than a directly observed scalar.
+
+## Reference Hallucination Distance estimator
+
+Because ECD is latent, the reference implementation estimates it as:
+
+```text
+HD = ω1 D_exec + ω2 M_omit + ω3 A_unsupported + ω4 C_contradict + ω5 E_calib
+```
+
+The implemented components are:
+
+| Component | Meaning |
+| --- | --- |
+| `D_exec` | Claim value disagrees with verified evidence for the same subject and predicate. |
+| `M_omit` | Claim provides no declared evidence lineage. |
+| `A_unsupported` | Claim cites evidence that is missing or unverified. |
+| `C_contradict` | Claims emit multiple values for the same subject and predicate. |
+| `E_calib` | Expressed confidence differs from observed claim support. |
+
+The implementation reports `(HD, Q_evidence)` instead of folding instrumentation quality into the metric:
+
+```text
+Q_evidence = |E_verified| / |E_total|
+```
+
+## Temporal dynamics
+
+Hallucination Delta is computed over deterministic ticks, not wall-clock time:
+
+```text
+HD_Δ = ΔHD / Δticks
+```
+
+Positive values indicate increasing structural divergence. Negative values indicate convergence toward evidence.
+
+## Reference implementation
+
+The current reference implementation is `verifiable/ecd.py`. It is deterministic, uses integer parts-per-million confidence values, represents scores as exact rational numbers, and can hash an estimator result into the existing AEGIS-Ω `LineageChain` for reproducible witness generation.
+
+Run the checks with:
+
+```bash
+python3 verifiable/test_ecd.py
+```
+
+The tests cover perfect alignment, controlled unsupported/contradictory claim insertion, deterministic witness hashing, tamper evidence, and positive Hallucination Delta under evidence divergence.
+
+## Evidence–Reasoning Benchmark protocol
+
+ERB should evaluate Hallucination Distance over trajectories containing:
+
+- observable claims;
+- evidence graph nodes;
+- deterministic sequence ticks;
+- induced perturbations;
+- reference labels.
+
+The benchmark has three tracks:
+
+1. **Track A:** estimate Hallucination Distance.
+2. **Track B:** predict Hallucination Delta across long execution horizons.
+3. **Track C:** infer structural evidence dependencies.
+
+## Metrological properties
+
+Any estimator in the HD family should satisfy:
+
+- **Sensitivity:** small structural perturbations produce measurable changes.
+- **Monotonicity:** increasing corruption should not decrease measured divergence.
+- **Repeatability:** independent implementations produce statistically equivalent estimates over identical evidence.
+- **Observer invariance:** equivalent evidence graphs yield equivalent measurements regardless of implementation.
+
+## Scope and limitations
+
+This framework measures claim/evidence alignment. It does not directly measure truthfulness, intent, consciousness, or internal reasoning. Estimator behavior depends on the graph distance, evidence quality depends on instrumentation, and weights require empirical calibration before promotion beyond T2.

verifiable/README.md

@@ -14,6 +14,8 @@ one.
 | `chain.py` | The domain-agnostic envelope — `canon()` (RFC 8785 → bytes, rejects float), `sha256_hex()`, `StageRecord`, `LineageChain` (append / `terminal_hash` / `certify`). The genomics proof inlines this for zero-dependency portability; here it is shared infra. |
 | `compliance_pipeline.py` | A **regulated decision-audit** pipeline (`INTAKE → EXTRACT → SCORE → DECISION`) — AEGIS's stated market: EU AI Act Article 12 tamper-evident decision records. Integer scorecard, adverse-action reason codes, integer threshold. |
 | `test_generality.py` | The proof. Exit 0 = all four claims hold. |
+| `ecd.py` | Deterministic reference estimator for Evidence–Claim Divergence / Hallucination Distance over claims and evidence nodes. |
+| `test_ecd.py` | Estimator checks for alignment, unsupported claims, contradiction, deterministic witness hashing, tamper evidence, and Hallucination Delta. |
 
 ## What is proven
 

verifiable/ecd.py

@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+"""
+Evidence–Claim Divergence reference estimator (T2 research instrument).
+
+This module gives the ECD / Hallucination Distance paper a small, deterministic,
+stdlib-only reference implementation over observable claims and evidence nodes. It
+is intentionally conservative: it measures claim/evidence alignment, not truth,
+intent, or model quality.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from fractions import Fraction
+from typing import Iterable
+
+from chain import LineageChain
+
+
+@dataclass(frozen=True)
+class Claim:
+    claim_id: str
+    subject: str
+    predicate: str
+    value: str
+    confidence_ppm: int = 1_000_000
+    evidence_ids: tuple[str, ...] = ()
+
+
+@dataclass(frozen=True)
+class EvidenceNode:
+    evidence_id: str
+    subject: str
+    predicate: str
+    value: str
+    verified: bool = True
+
+
+@dataclass(frozen=True)
+class HDWeights:
+    execution: int = 1
+    omission: int = 1
+    unsupported: int = 1
+    contradiction: int = 1
+    calibration: int = 1
+
+
+@dataclass(frozen=True)
+class HDResult:
+    execution_divergence: Fraction
+    evidence_omission: Fraction
+    unsupported_assertions: Fraction
+    contradictory_claims: Fraction
+    calibration_mismatch: Fraction
+    hallucination_distance: Fraction
+    evidence_quality: Fraction
+    verified_evidence: int
+    total_evidence: int
+
+    def as_payload(self) -> dict:
+        return {
+            "execution_divergence_ppm": ppm(self.execution_divergence),
+            "evidence_omission_ppm": ppm(self.evidence_omission),
+            "unsupported_assertions_ppm": ppm(self.unsupported_assertions),
+            "contradictory_claims_ppm": ppm(self.contradictory_claims),
+            "calibration_mismatch_ppm": ppm(self.calibration_mismatch),
+            "hallucination_distance_ppm": ppm(self.hallucination_distance),
+            "evidence_quality_ppm": ppm(self.evidence_quality),
+            "verified_evidence": self.verified_evidence,
+            "total_evidence": self.total_evidence,
+        }
+
+
+def ppm(x: Fraction) -> int:
+    return int(x * 1_000_000)
+
+
+def _claim_key(claim: Claim) -> tuple[str, str]:
+    return claim.subject, claim.predicate
+
+
+def _evidence_key(node: EvidenceNode) -> tuple[str, str]:
+    return node.subject, node.predicate
+
+
+def estimate_hd(
+    claims: Iterable[Claim],
+    evidence: Iterable[EvidenceNode],
+    weights: HDWeights = HDWeights(),
+) -> HDResult:
+    claim_list = sorted(claims, key=lambda c: c.claim_id)
+    evidence_list = sorted(evidence, key=lambda e: e.evidence_id)
+    total_claims = len(claim_list)
+    total_evidence = len(evidence_list)
+    verified_evidence = sum(1 for node in evidence_list if node.verified)
+
+    if total_claims == 0:
+        zero = Fraction(0, 1)
+        quality = Fraction(verified_evidence, total_evidence) if total_evidence else zero
+        return HDResult(zero, zero, zero, zero, zero, zero, quality, verified_evidence, total_evidence)
+
+    evidence_by_id = {node.evidence_id: node for node in evidence_list}
+    verified_by_key: dict[tuple[str, str], list[EvidenceNode]] = {}
+    for node in evidence_list:
+        if node.verified:
+            verified_by_key.setdefault(_evidence_key(node), []).append(node)
+
+    execution_mismatches = 0
+    omissions = 0
+    unsupported = 0
+    calibration_error = Fraction(0, 1)
+    seen_values: dict[tuple[str, str], set[str]] = {}
+
+    for claim in claim_list:
+        key = _claim_key(claim)
+        supporting = [evidence_by_id[eid] for eid in claim.evidence_ids if eid in evidence_by_id and evidence_by_id[eid].verified]
+        candidates = supporting or verified_by_key.get(key, [])
+        matching = [node for node in candidates if _evidence_key(node) == key and node.value == claim.value]
+
+        if not claim.evidence_ids:
+            omissions += 1
+        elif not supporting:
+            unsupported += 1
+        if not matching:
+            execution_mismatches += 1
+
+        observed_correct = Fraction(1, 1) if matching else Fraction(0, 1)
+        expressed = Fraction(claim.confidence_ppm, 1_000_000)
+        calibration_error += abs(expressed - observed_correct)
+        seen_values.setdefault(key, set()).add(claim.value)
+
+    contradictory_groups = sum(1 for values in seen_values.values() if len(values) > 1)
+    possible_groups = len(seen_values) or 1
+
+    execution = Fraction(execution_mismatches, total_claims)
+    omission = Fraction(omissions, total_claims)
+    unsupported_rate = Fraction(unsupported, total_claims)
+    contradiction = Fraction(contradictory_groups, possible_groups)
+    calibration = calibration_error / total_claims
+    quality = Fraction(verified_evidence, total_evidence) if total_evidence else Fraction(0, 1)
+
+    weight_total = weights.execution + weights.omission + weights.unsupported + weights.contradiction + weights.calibration
+    hd = (
+        weights.execution * execution
+        + weights.omission * omission
+        + weights.unsupported * unsupported_rate
+        + weights.contradiction * contradiction
+        + weights.calibration * calibration
+    ) / weight_total
+
+    return HDResult(execution, omission, unsupported_rate, contradiction, calibration, hd, quality, verified_evidence, total_evidence)
+
+
+def hd_delta(previous: HDResult, current: HDResult, elapsed_ticks: int) -> Fraction:
+    if elapsed_ticks <= 0:
+        raise ValueError("elapsed_ticks must be positive")
+    return (current.hallucination_distance - previous.hallucination_distance) / elapsed_ticks
+
+
+def measurement_chain(result: HDResult) -> LineageChain:
+    chain = LineageChain()
+    chain.append("ECD_ESTIMATE", result.as_payload())
+    return chain

verifiable/test_ecd.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+"""Deterministic checks for the Evidence–Claim Divergence estimator."""
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+
+from ecd import Claim, EvidenceNode, estimate_hd, hd_delta, measurement_chain, ppm
+
+
+def test_perfect_alignment_has_zero_hd_and_full_quality():
+    evidence = [EvidenceNode("e1", "run", "exit_code", "0")]
+    claims = [Claim("c1", "run", "exit_code", "0", evidence_ids=("e1",))]
+    result = estimate_hd(claims, evidence)
+    assert result.hallucination_distance == 0
+    assert result.evidence_quality == 1
+
+
+def test_unsupported_and_contradictory_claims_increase_hd():
+    evidence = [EvidenceNode("e1", "run", "exit_code", "0")]
+    clean = estimate_hd([Claim("c1", "run", "exit_code", "0", evidence_ids=("e1",))], evidence)
+    corrupted = estimate_hd([
+        Claim("c1", "run", "exit_code", "0", evidence_ids=("e1",)),
+        Claim("c2", "run", "exit_code", "1", confidence_ppm=900_000, evidence_ids=("missing",)),
+    ], evidence)
+    assert corrupted.hallucination_distance > clean.hallucination_distance
+    assert corrupted.unsupported_assertions > 0
+    assert corrupted.contradictory_claims > 0
+
+
+def test_measurement_chain_is_reproducible_and_tamper_evident():
+    result = estimate_hd([Claim("c1", "run", "exit_code", "0", evidence_ids=("e1",))], [EvidenceNode("e1", "run", "exit_code", "0")])
+    chain_a = measurement_chain(result)
+    chain_b = measurement_chain(result)
+    assert chain_a.terminal_hash() == chain_b.terminal_hash()
+    chain_a.records[0].output["hallucination_distance_ppm"] = 1
+    assert chain_a.certify()["is_valid"] is False
+
+
+def test_hd_delta_uses_deterministic_ticks():
+    evidence = [EvidenceNode("e1", "run", "exit_code", "0")]
+    before = estimate_hd([Claim("c1", "run", "exit_code", "0", evidence_ids=("e1",))], evidence)
+    after = estimate_hd([Claim("c1", "run", "exit_code", "1", confidence_ppm=1_000_000, evidence_ids=("e1",))], evidence)
+    assert ppm(hd_delta(before, after, elapsed_ticks=2)) > 0
+
+
+if __name__ == "__main__":
+    tests = [
+        test_perfect_alignment_has_zero_hd_and_full_quality,
+        test_unsupported_and_contradictory_claims_increase_hd,
+        test_measurement_chain_is_reproducible_and_tamper_evident,
+        test_hd_delta_uses_deterministic_ticks,
+    ]
+    for test in tests:
+        test()
+    print(f"ECD estimator checks passed: {len(tests)}")