Agent-based fact verification using LLM reasoning over PubMed results

[sarvanithin]

Overview

The current FactVerifier uses keyword matching to score PubMed abstracts against claims. This works but misses semantic nuance. This issue tracks upgrading it to use an LLM agent that reasons over retrieved abstracts.

Current behavior

FactVerifier.verify(claim) → keyword overlap scoring → confidence score

Proposed behavior

claim → PubMed search → top 5 abstracts retrieved
     → LLM agent prompt: "Given these abstracts, is the claim supported, contradicted, or inconclusive?"
     → structured verdict: {verdict, confidence, citations}

Implementation hints

• FactVerifier already has the PubMed retrieval pipeline in medguard/knowledge/pubmed.py
• Add AgentFactVerifier class in medguard/guardrails/fact_check.py that accepts an LLMCallerProtocol
• Prompt should include abstract text + claim, ask for JSON verdict
• Should be opt-in (requires LLM caller)
• Keep the existing keyword-based verifier as fallback when no LLM is configured

Files to modify

• medguard/guardrails/fact_check.py — add AgentFactVerifier
• medguard/config.py — add use_agent: bool = False to FactCheckConfig
• medguard/core.py — pass llm_caller to FactVerifier when use_agent=True

Acceptance criteria

• AgentFactVerifier.verify(claim) returns a FactEvidence with reasoning field
• Falls back to keyword verifier when no LLM is configured
• Integration test against real PubMed + real LLM (marked @pytest.mark.integration)
• Ruff clean, all existing tests pass