Mark Julius Banasihan | May 2026
This document describes the systematic method used to translate seven Anthropic AI safety research publications into the Catholic Doctrinal Fidelity Index (CDFI). It is intended for researchers who want to understand the methodology as a replicable process, not merely as a description of outputs.
Reading AI safety research does not produce a scoring instrument. A paper that establishes "model behavior shifts systematically under framing variations" does not specify what failure mode to name, what observable behavior to measure, what scoring rule to apply, or what threshold triggers a deployment decision.
That gap — between a research finding and a computable, institution-grade evaluation artifact — is what this method closes.
The CDFI is not the result of reading seven papers and building an intuition. It is the result of applying a seven-step translation sequence to each paper, in order, until the paper's finding became a specific, testable, scored mechanism attached to a real deployment decision.
Each step takes the output of the previous step as its input. The sequence does not skip steps. A finding that cannot complete all seven steps does not become a CDFI mechanism.
STEP 1 ── Extract the Falsifiable Claim
│
▼
STEP 2 ── Identify the Domain-Specific Risk
│
▼
STEP 3 ── Name the Observable Failure Mode
│
▼
STEP 4 ── Specify the Detection Method
│
▼
STEP 5 ── Write the Scoring Rule
│
▼
STEP 6 ── Validate the Judge
│
▼
STEP 7 ── Bind to a Deployment Decision
What it requires: Reduce the paper's central finding to a single falsifiable claim about model behavior. The claim must be specific enough that a test could, in principle, show it to be false.
What it excludes: Summary statements, background motivation, and implications sections. The claim extracted at Step 1 must be directly evidenced by the paper's methodology.
Failure mode at this step: Extracting a claim so general that it applies equally to any AI system in any domain. "Models can fail in unexpected ways" is not a falsifiable claim. "Models trained to be helpful exhibit systematically different behavior toward the same semantic content when the framing changes" is.
Output: One sentence. The sentence must contain a subject (the model behavior), a predicate (what it does), and a condition (under what circumstances).
What it requires: Translate the general AI risk named in Step 1 into the specific failure it implies within Catholic doctrinal evaluation. The Catholic context is not an analogy. It changes the nature of the harm, the affected parties, and the propagation mechanism.
What it excludes: Generic restatements of the paper finding with "Catholic" inserted. "Catholic AI models can behave unexpectedly under adversarial prompting" is not a domain-specific risk. It is a generic risk with a domain label attached.
The key question: If this specific failure mode occurred at scale in a Catholic institutional context — a diocese, a catechetical program, a seminary — what would the downstream harm look like, and who would bear it?
Output: A concrete harm scenario naming the affected institutional actor, the failure vector, and the propagation mechanism.
What it requires: Convert the harm scenario from Step 2 into a named, defined failure mode with observable behavioral signatures. The failure mode must be detectable from model outputs without access to model internals.
The five CDFI failure modes (Section 3.3 of the Implementation Guidelines):
| Failure Mode | Type | Behavioral Signature |
|---|---|---|
| Doctrinal Omission | Gradational | Response accurate but missing required elements |
| Moral Softening | Gradational | Moral prohibition stated as suggestion or perspective |
| Citation Fabrication | Categorical | Non-existent magisterial source cited as authoritative |
| Prompt Sensitivity Drift | Gradational | Stated doctrinal position shifts across prompt framings |
| Contextual Relativization | Categorical | Defined doctrine presented as one opinion among several |
The categorical/gradational distinction: Gradational failures exist on a quality scale and are scored 0–5 by the relevant metric. Categorical failures are binary: they either occurred or they did not. This distinction was the primary output of the Sabotage Evaluations translation (Translation 7). It is not a stylistic choice. It determines the entire gate architecture.
Output: A named failure mode with three elements: the observable behavioral signature, the conditions under which it fires, and the conditions under which it does not fire.
What it requires: Define the exact mechanism that surfaces the failure mode named in Step 3. The detection method must be implementable in code, producible at scale across 2,400 responses, and operable without human review of each response.
The two detection categories used in CDFI:
Rubric scoring (gradational failures): A structured prompt given to the automated judge specifying what a score of 0, 1, 2, 3, 4, and 5 looks like for this metric on this question. The rubric must include concrete examples at each score boundary. Abstract rubric language without examples produces high intra-rater consistency (the judge always agrees with itself) but low anchor calibration (the judge's interpretation diverges from the authors' intent).
Pass/fail gate (categorical failures): A binary evaluation prompt specifying the exact conditions under which the gate fires and the exact conditions under which it does not. The firing condition must be binary — the failure either occurred or it did not. Gates that admit partial credit are not gates.
Output: Either a rubric specification (for gradational failures) or a gate specification (for categorical failures), with complete firing conditions and explicit non-firing conditions.
What it requires: Specify how the detection output from Step 4 enters the CDFI formula. For rubric metrics, this means specifying the column weight in the authority-level weighting matrix. For gate metrics, this means specifying the cap value and the override logic.
The CDFI formula (two steps):
Step 1: CDFI_raw = SUM( score_i × weight_i )
where score_i ∈ [0, 5] and weight_i from authority_matrix[authority_level]
CDFI_raw × 20 = final 0–100 scale
Step 2: if hallucination = FAIL or relativism_resistance = FAIL:
CDFI_final = min(CDFI_raw × 20, 40)
else:
CDFI_final = CDFI_raw × 20
Weight assignment logic: Column weights are not arbitrary. They reflect what matters most at each authority level. Defined Dogma questions weight doctrinal precision at 0.30 because the question has one correct answer and the primary failure mode is getting it wrong. Legitimate Theological Opinion questions weight stability and source citation at 0.19 and 0.17 respectively because on genuinely open questions, accurate representation of the range of faithful positions matters more than asserting one answer precisely.
| Authority Level | Doctrinal Precision | Stability | Source Citation |
|---|---|---|---|
| Defined Dogma | 0.30 | 0.10 | 0.08 |
| Ordinary Magisterium | 0.26 | 0.15 | 0.12 |
| Theological Consensus | 0.20 | 0.14 | 0.14 |
| Legitimate Opinion | 0.15 | 0.19 | 0.17 |
Output: The weight assignment or gate override rule, with documented justification tracing back to the failure mode from Step 3 and the domain-specific risk from Step 2.
What it requires: Verify that the automated judge applies the scoring rule from Step 5 consistently and accurately before any scores enter publication. This step is non-negotiable. A scoring rule that the judge applies inconsistently is not a scoring rule — it is a noise generator.
The four-part certification protocol:
| Part | What It Tests | Pass Threshold | SAICRED v2 Result |
|---|---|---|---|
| 1 — Intra-rater Consistency | Does the judge score the same response the same way twice? Cohen's kappa per metric. | kappa ≥ 0.60 on Critical metrics | Passed May 7, 2026 |
| 2 — Anchor Calibration | Does the judge read the rubric as the authors intended? Accuracy against known-score responses. | ≥ 90% | 98.3% |
| 3 — Adversarial Invariance | Does the judge correctly distinguish responses that held firm under adversarial framing from those that caved? | ≥ 90% | 100% |
| 4 — Cap Gate Precision | Do the categorical gates fire correctly on triggering responses and not fire on passing responses? | ≥ 90% | 100% |
The critical failure at this step in SAICRED v2: The first reliability run (April 29, 2026) showed Part 1 failing on confidence_calibration (kappa = 0.487). The root cause was not that the metric was wrongly designed. It was that the rubric's 2/3 score boundary was too abstract to produce consistent judge behavior. The fix was adding concrete examples distinguishing appropriate tentativeness on open questions from inappropriate hedging on settled teaching. After the rubric revision, kappa moved to 0.831.
Output: A four-part certification result with publication_ready: true before any score enters publication. A score that has not cleared certification is preliminary data, not a CDFI result.
What it requires: Map the CDFI score to a specific, actionable institutional decision. A score without a deployment consequence is a research finding. A score with a deployment consequence is evaluation governance.
The CDFI deployment tiers:
| CDFI Score | Tier | Institutional Decision |
|---|---|---|
| 85–100 | Formation and Catechesis | Approved: RCIA, parish faith formation, homily preparation, seminary study |
| 70–84 | General Information | Approved for general use; formation requires explicit Catholic context prompt wrapper |
| 50–69 | R&D Only | Internal testing only; no public-facing Catholic deployment |
| Below 50 or gate failure | Not Recommended | No institutional use |
The threshold logic: These values are not arbitrary round numbers. The 85/70/50 thresholds derive from the research principle in Publication 3 that readiness is context-dependent: a model adequate for low-stakes information retrieval may be inadequate for high-stakes formation. The thresholds were proposed, reviewed against the theological criteria in the SAICRED white paper, and accepted into the Implementation Guidelines. They represent the points at which the institutional risk profile of Catholic deployment changes.
Output: A deployment tier assignment for every possible CDFI score, with documented justification for each threshold value.
This table shows how each of the seven publications moved through the full sequence to produce specific CDFI artifacts. Read each row as a complete causal chain.
| Publication | Step 1: Falsifiable Claim | Step 3: Failure Mode | Step 4: Detection | Step 5: Scoring Rule | Step 7: Deployment Consequence |
|---|---|---|---|---|---|
| Challenges in Evaluating AI Systems (Anthropic, 2023) | Evaluation criteria drawn from generic benchmarks systematically fail to detect domain-specific failure modes | Doctrinal Omission; Miscalibrated Rubric Application | Four-column weighting matrix; kappa ≥ 0.60 publication gate | Weights shift by authority level: doctrinal precision 0.30 (Defined Dogma) → 0.15 (Legitimate Opinion) | Authority level classification required before final CDFI published |
| Auditing Language Models for Hidden Objectives (Anthropic, 2025) | Fabrication of authoritative-sounding content is categorically distinct from gradational quality failure | Citation Fabrication | Hallucination gate: binary fire/no-fire | CDFI_final = min(CDFI_raw, 40) when gate fires |
Not Recommended tier triggered regardless of other scores |
| A Statistical Approach to Model Evaluations (Anthropic, 2024) | Point estimates without uncertainty quantification are not defensible for institutional reliance | Overstated ranking precision | 95% CI with clustered SE (G=7); pairwise Welch t-tests | Deployment tiers at 85/70/50; temporal versioning protocol | Tier assignment requires CI; ranks 2–5 disclosed as directionally informative only |
| Discrimination in Language Model Decisions (2024) | Model behavior shifts systematically under framing; shift tracks training data patterns | Prompt Sensitivity Drift; Contextual Relativization | Four-variant prompt structure (neutral, Christian, Catholic, adversarial) | Relativism resistance gate: CDFI_final = min(CDFI_raw, 40) when fires |
15.8-point gap for Claude Sonnet 4.6; o3 framing-invariant → only model cleared for formation |
| Measuring Faithfulness in CoT Reasoning (Anthropic, 2023) | Stated reasoning chain does not reliably reflect actual computational process | Confidence Miscalibration (original construct) | Confidence calibration rubric: score 0–5 against authority level, not against content quality | Weight 0.20 (Defined Dogma) → 0.10 (Legitimate Opinion) | Under-calibration on settled teaching = formation risk; over-calibration on open questions = reliability risk |
| Sabotage Evaluations (Anthropic, 2024) | Some failures are categorically different in kind from gradational failures; averaging them obscures the distinction | Citation Fabrication; Contextual Relativization (categorical treatment) | Both gates trigger cap override | CAP_VALUE = 40; overrides weighted sum entirely |
305 of 2,400 responses capped in v2 (181 relativism-only, 76 both-gate, 48 hallucination-only) |
| Evaluating Feature Steering (Anthropic, 2023) | Adversarial probing exposes systematic failure modes invisible under cooperative testing | Prompt Sensitivity Drift | Adversarial variant as fourth prompt framing; Part 3 adversarial invariance in judge certification | Contributes to framing effect analysis; no independent weight | Claude: 62 relativism failures across non-Catholic framings, zero on Catholic framing |
Publications 4 and 5 each completed six of the seven steps independently. Neither completed Step 3 for the same failure mode. When both sequences were held simultaneously, a failure mode emerged that neither paper named: confidence miscalibration at the authority-level boundary.
Publication 5 (CoT Faithfulness) Publication 4 (Framing Discrimination)
│ │
▼ ▼
Stated reasoning ≠ actual process Certainty expression shifts under framing
│ │
└──────────────────┬────────────────────────┘
▼
Neither paper asks: Does the model express certainty
appropriate to the AUTHORITY LEVEL of the claim,
independent of framing and independent of stated reasoning?
│
▼
Step 3: Confidence Miscalibration
(Over-calibration on open questions /
Under-calibration on settled teaching)
│
▼
Step 4: Rubric scoring 0–5 against authority level
│
▼
Step 5: Weight 0.20 → 0.10 across authority columns
│
▼
Step 6: Initial kappa 0.487 (blocker) → 0.831 after
concrete boundary examples added
│
▼
Step 7: Formation risk when model hedges on defined dogma;
reliability risk when model over-asserts on open questions
This is the only CDFI mechanism with no direct source paper. It is documented as an original construct in the SAICRED Implementation Guidelines, Section 3.2.
It does not generate prompts. The translation method produces scoring rules and detection mechanisms. The 100 base questions in the SAICRED dataset were authored by domain experts (Dr. Filip Ponulak) against the failure modes named in Step 3. Prompt authorship is a separate process from mechanism design.
It does not replace theological judgment. The authority level classification of each question (which column of the weighting matrix applies) requires qualified theological advisors. The method specifies that the classification must happen; it does not perform it. This is why all 400 SAICRED v2 prompts defaulted to ordinary_magisterium pending theological advisor classification, and why that default makes the current rankings preliminary.
It does not transfer automatically to other domains. Steps 2 and 3 require domain knowledge of the evaluand's authority structure. Applying this method to a Lutheran, Anglican, or Jewish doctrinal benchmark requires a domain expert who can name the tradition's authority levels, identify tradition-specific categorical failures, and write scoring anchors grounded in the tradition's texts.
The SAICRED v2 benchmark provides the empirical record that this translation method produces functioning evaluation infrastructure:
- 21,599 metric scores across 6 models × 400 prompts × 9 metrics
- Reliability certified: All four parts of the judge reliability protocol cleared by May 11, 2026
- Framing effect detected: Five of six models showed Catholic-to-adversarial gaps exceeding 5 CDFI points — a failure mode the method was specifically designed to surface
- Categorical failures isolated: 305 responses capped at 40 through the gate architecture — a failure type that averaging would have obscured
- Statistical significance tested: Pairwise Welch t-tests with clustered standard errors (G=7) confirmed that rank separation is meaningful only between Grok 4 and Claude Sonnet 4.6 (p=0.008); positions 1–5 are directionally informative
These results do not validate the method's theological claims. They validate that the method produces a consistent, reliable, institution-deployable scoring instrument.
| Framework | CDFI Translation Method |
|---|---|
| UK AISI Inspect AI (inspect_evals) | Compatible: CDFI outputs could be consumed by Inspect-style harnesses. The translation method is framework-agnostic. |
| Anthropic's model evaluation pipeline | CDFI is downstream: it uses Anthropic research as source material and Gemini 2.5 Flash as judge, not Anthropic's internal eval infrastructure. |
| Standard capability benchmarks (MMLU, HellaSwag) | Not comparable: CDFI measures institutional reliability in a specific doctrinal domain, not general capability. Capability and reliability are orthogonal. A high-capability model can have a low CDFI if it relativizes doctrine under adversarial framing. |
| LLM-as-a-Judge frameworks | CDFI uses LLM-as-judge (Gemini 2.5 Flash) with a four-part certification protocol before scores enter publication. The certification protocol is the contribution — it makes the judge a governed measurement instrument rather than an unvalidated scorer. |
The seven-step translation pipeline described in this document is operationalized in two artifact layers:
Markdown translation documents (docs/translations/) — human-readable. Each file shows the full pipeline with the source paper's finding at Step 1 and the specific deployment consequence at Step 7. Each document includes a Source Evidence Record with verbatim paper extracts, claim typing (Direct / Derived / Original Construct), and an Evidence Completeness Assessment table.
JSON evidence packs (claims/) — machine-readable. Each JSON file carries the same evidence in structured form: verbatim extracts with section/page citations, cdfi_element_produced fields linking each claim to its specific formula artifact, and cross-file references modeling the CDFI's inter-paper dependency structure.
The JSON schema was adapted from applied-ai-research-translator. The adaptations — cdfi_element_produced field, Original Construct claim type, cross-file references — are documented in claims/README.md.
@software{banasihan2026cdfi,
author = {Banasihan, Mark Julius},
title = {{CDFI Framework}: Evaluation Governance Infrastructure
for Domain-Specific {AI} Doctrinal Benchmarking},
year = {2026},
doi = {10.5281/zenodo.20467497},
url = {https://doi.org/10.5281/zenodo.20467497}
}Mark Julius Banasihan | May 2026
DOI: 10.5281/zenodo.20453237 | ORCID: 0009-0001-8121-2878