Add Evidence–Claim Divergence (ECD) estimator, docs, and tests - #239
Add Evidence–Claim Divergence (ECD) estimator, docs, and tests#239Tarik Skalić (tarikskalic33) wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
aegisomega | 93d4204 | Commit Preview URL Branch Preview URL |
Jul 28 2026, 03:59 PM |
ghost
mentioned this pull request
PR Summary by QodoAdd deterministic Evidence–Claim Divergence estimator with docs and tests
AI Description
Diagram
High-Level Assessment
Files changed (4)
|
Code Review by Qodo
Context used✅ Compliance rules (platform):
11 rules 1. Unchecked weight_total division
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
| weight_total = weights.execution + weights.omission + weights.unsupported + weights.contradiction + weights.calibration | ||
| hd = ( | ||
| weights.execution * execution | ||
| + weights.omission * omission | ||
| + weights.unsupported * unsupported_rate | ||
| + weights.contradiction * contradiction | ||
| + weights.calibration * calibration | ||
| ) / weight_total |
There was a problem hiding this comment.
1. Unchecked weight_total division 🐞 Bug ≡ Correctness
In estimate_hd(), caller-supplied HDWeights can sum to 0 (or include negative values), causing a ZeroDivisionError or semantically invalid/negative HD output. This makes the estimator fragile for any downstream experiment that tunes weights.
Agent Prompt
### Issue description
`estimate_hd()` divides by `weight_total` without validating it. If weights sum to 0, the function raises `ZeroDivisionError`; if weights are negative, the returned HD can be negative or otherwise outside the intended scale.
### Issue Context
Weights are part of the public API and are likely to be tuned by downstream users.
### Fix Focus Areas
- verifiable/ecd.py[39-45]
- verifiable/ecd.py[141-148]
### Suggested fix
- Add validation (either in `HDWeights.__post_init__()` or at the start of `estimate_hd()`):
- require all weights to be non-negative integers (or at least `>= 0`)
- require `weight_total > 0`
- raise `ValueError` with a clear message if invalid.
- (Optional) Add a small test that passing all-zero weights raises `ValueError` (instead of crashing with `ZeroDivisionError`).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| claim_list = sorted(claims, key=lambda c: c.claim_id) | ||
| evidence_list = sorted(evidence, key=lambda e: e.evidence_id) | ||
| total_claims = len(claim_list) | ||
| total_evidence = len(evidence_list) | ||
| verified_evidence = sum(1 for node in evidence_list if node.verified) | ||
|
|
||
| if total_claims == 0: | ||
| zero = Fraction(0, 1) | ||
| quality = Fraction(verified_evidence, total_evidence) if total_evidence else zero | ||
| return HDResult(zero, zero, zero, zero, zero, zero, quality, verified_evidence, total_evidence) | ||
|
|
||
| evidence_by_id = {node.evidence_id: node for node in evidence_list} | ||
| verified_by_key: dict[tuple[str, str], list[EvidenceNode]] = {} | ||
| for node in evidence_list: | ||
| if node.verified: | ||
| verified_by_key.setdefault(_evidence_key(node), []).append(node) | ||
|
|
There was a problem hiding this comment.
2. Duplicate evidence ids overwrite 🐞 Bug ☼ Reliability
estimate_hd() silently overwrites earlier EvidenceNode entries when multiple nodes share the same evidence_id, which can change which evidence is treated as "supporting" and therefore change HD results. Because ties are not rejected, results can become sensitive to the iteration order of malformed inputs (e.g., when evidence is supplied from an unordered iterable) despite the module’s deterministic intent.
Agent Prompt
### Issue description
`evidence_by_id` is built with a dict comprehension keyed by `evidence_id`, so duplicates are silently dropped (last one wins). This is a data-integrity hazard and can make outputs depend on which duplicate happens to be retained.
### Issue Context
The module explicitly positions itself as a deterministic reference estimator, so ambiguous identifiers should be rejected early and loudly.
### Fix Focus Areas
- verifiable/ecd.py[5-8]
- verifiable/ecd.py[90-106]
### Suggested fix
- Validate uniqueness:
- after sorting, scan `claim_list` for duplicate `claim_id` and raise `ValueError` if found
- scan `evidence_list` for duplicate `evidence_id` and raise `ValueError` if found
- (Alternative) If duplicates are intended, change `evidence_by_id` to map IDs to a list and define a deterministic selection rule; but rejection is usually better for a measurement substrate.
- Add a targeted test: duplicate evidence IDs should raise `ValueError`.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| observed_correct = Fraction(1, 1) if matching else Fraction(0, 1) | ||
| expressed = Fraction(claim.confidence_ppm, 1_000_000) | ||
| calibration_error += abs(expressed - observed_correct) |
There was a problem hiding this comment.
3. Out-of-range confidence accepted 🐞 Bug ≡ Correctness
estimate_hd() converts Claim.confidence_ppm directly into a Fraction with no bounds checks, so negative values or values > 1_000_000 produce expressed confidence outside [0,1]. This can inflate calibration_mismatch (and overall HD) beyond the intended normalized scale, making results hard to interpret and compare.
Agent Prompt
### Issue description
`Claim.confidence_ppm` is treated as parts-per-million probability, but it is not validated. Out-of-range values yield expressed confidence outside [0,1], distorting calibration and HD.
### Issue Context
The docs and code use PPM as a deterministic probability representation; that implies a fixed domain of valid values.
### Fix Focus Areas
- verifiable/ecd.py[19-27]
- verifiable/ecd.py[126-128]
### Suggested fix
- Add validation (preferably in `Claim.__post_init__()`):
- require `0 <= confidence_ppm <= 1_000_000`
- raise `ValueError` on violation
- Add a small test case that constructing a claim with `confidence_ppm=-1` or `1_000_001` raises `ValueError`.
- (If you explicitly want to allow out-of-range), document that behavior and its impact on calibration/HD scaling.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93d420468c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| elif not supporting: | ||
| unsupported += 1 |
There was a problem hiding this comment.
When a claim cites both one valid verified evidence id and another missing or unverified id, supporting is non-empty, so this branch leaves unsupported_assertions at zero and the HD can still report perfect alignment. The new estimator docs define A_unsupported as a claim citing evidence that is missing or unverified (docs/EVIDENCE_CLAIM_DIVERGENCE.md:41), so a bad lineage entry can currently be hidden just by appending any good evidence id; check each cited id rather than only whether any support exists.
Useful? React with 👍 / 👎.
| | `compliance_pipeline.py` | A **regulated decision-audit** pipeline (`INTAKE → EXTRACT → SCORE → DECISION`) — AEGIS's stated market: EU AI Act Article 12 tamper-evident decision records. Integer scorecard, adverse-action reason codes, integer threshold. | | ||
| | `test_generality.py` | The proof. Exit 0 = all four claims hold. | | ||
| | `ecd.py` | Deterministic reference estimator for Evidence–Claim Divergence / Hallucination Distance over claims and evidence nodes. | | ||
| | `test_ecd.py` | Estimator checks for alignment, unsupported claims, contradiction, deterministic witness hashing, tamper evidence, and Hallucination Delta. | |
There was a problem hiding this comment.
This adds test_ecd.py as the estimator check suite, but I checked the verifiable workflow (.github/workflows/verifiable-proofs.yml:50-87) and the session certifier proof list (verifiable/certify_all.py:41-47), and neither invokes it. As a result, PRs changing verifiable/ecd.py still get a green CI/session certificate even if these new checks fail; please add python3 test_ecd.py to the workflow and PROOFS list, updating the pinned session certificate as needed.
Useful? React with 👍 / 👎.
|
|
||
|
|
||
| def ppm(x: Fraction) -> int: | ||
| return int(x * 1_000_000) |
There was a problem hiding this comment.
For ERB-sized traces with more than 1,000,000 weighted opportunities, a real non-zero divergence below one ppm is floored to 0 here before measurement_chain() hashes as_payload(). That lets distinct exact Fraction results, including clean versus one tiny mismatch, produce identical witness payloads despite the estimator keeping exact rationals; serialize numerator/denominator pairs or another lossless form for chain payloads.
Useful? React with 👍 / 👎.
| execution_mismatches += 1 | ||
|
|
||
| observed_correct = Fraction(1, 1) if matching else Fraction(0, 1) | ||
| expressed = Fraction(claim.confidence_ppm, 1_000_000) |
There was a problem hiding this comment.
When a caller supplies an impossible confidence_ppm such as 1_500_000 or a negative value, this code treats it as a valid confidence fraction instead of rejecting it. Since the docs define confidence as integer parts-per-million, out-of-range inputs can create calibration mismatches above 1 or penalize a fully supported claim even though the claim/evidence pair is aligned; validate 0 <= confidence_ppm <= 1_000_000 before using it in the estimator.
Useful? React with 👍 / 👎.
1 participant
Motivation
LineageChainfor deterministic auditing.Description
verifiable/ecd.py, a stdlib-only, deterministic estimator implementing Claim and EvidenceNode dataclasses, weighted HD components (execution,omission,unsupported,contradiction,calibration), exact rational scoring viaFraction,hd_delta(tick-based), andmeasurement_chainthat emits aLineageChainwitness.verifiable/test_ecd.py, a small deterministic test harness that checks perfect alignment → zero HD, increases under unsupported/contradictory claims, reproducible/tamper-evident measurement chains, and positivehd_deltafor divergence.docs/EVIDENCE_CLAIM_DIVERGENCE.md, a user-facing document describing the formal objects (C, E), the reference HD estimator, ERB tracks, temporal dynamics, evidence quality reporting, metrological desiderata, and limitations.verifiable/README.mdto list the new estimator and tests so the verifiable substrate documents the added research instrument.Testing
python3 verifiable/test_ecd.py, which completed successfully (all tests passed).python3 verifiable/test_generality.py, which completed successfully.python3 -m py_compile verifiable/ecd.py verifiable/test_ecd.py, which succeeded with no syntax errors.Codex Task