Make identity adjudication authoritative before metadata merge and PDF fetch

[917Dhj]

Problem Statement

DeepPaperNote can currently accept metadata and PDF locations from a provider candidate before it has established that the candidate represents the same Paper Identity as the user's input or Zotero item. Metadata collection destructively merges provider records first, and the Identity Verification Boundary evaluates the already-merged result afterward. This loses field-level provenance and allows a same-title but different work to contribute a DOI, arXiv ID, abstract, source URL, or PDF URL.

The current safeguards address individual symptoms but do not establish one authoritative admission decision. In particular:

• A strong Zotero or DOI anchor can still retain a PDF location or secondary identifier from an unrelated title candidate.
• Title lookup normalization removes meaningful symbols, so distinct titles such as C++ and C, C# and C, or Na+ and Na can be treated as exact matches.
• A malformed optional Zotero parent or attachment URL can escape as an exception instead of rejecting only that source location and preserving valid Paper Identity evidence.
• PDF acquisition can perform metadata enrichment again after the Canonical Identity Artifact has been accepted, creating a second identity-selection path.
• Raw-input and standalone acquisition paths can bypass the canonical identity decision.

Passing general tests is therefore insufficient: the pipeline must make identity admission authoritative before any candidate metadata or source location reaches PDF fetch or model-facing writing input.

Solution

Preserve the existing single-paper stage order and make the current identity-contract stage the one authoritative Identity Verification Boundary.

Metadata providers and Zotero will produce separate, provenance-bearing Identity Evidence observations instead of directly merging identity-bearing fields. The identity-contract stage will adjudicate the input anchor and all observations, then emit an upgraded Canonical Identity Artifact containing the accepted Work-Level Identity, accepted descriptive metadata, ordered and identity-bound Source Manifestations, rejected observations with reasons, the Identity Verdict, and the Identity Repair Trace.

Only observations that have been accepted as the same Paper Identity or an explicitly equivalent Paper Manifestation may contribute metadata or fetchable source locations. Strong identifier conflicts fail closed. Title-only similarity may retrieve candidates but may not canonicalize them. When no shared Strong Identity Anchor exists, automatic acceptance requires a symbol-preserving title match plus independent normalized leading-author and compatible-year evidence; abstract or first-page similarity may support the decision but may not make a title-only match sufficient.

PDF acquisition becomes transport-only. It consumes ordered bound sources from an accepted Canonical Identity Artifact, verifies downloaded bytes, records attempts, and never performs metadata enrichment or identity repair. Synthesis consumes accepted metadata rather than the raw provider collection artifact.

User Stories

1. As a DeepPaperNote user, I want the run to remain anchored to the paper I supplied, so that a same-title work cannot silently replace it.
2. As a Zotero user, I want a unique Zotero item and its attachment relationship to remain authoritative, so that web enrichment cannot redirect the run to another paper.
3. As a DOI user, I want conflicting provider identifiers to fail closed, so that the note is never written about an uncertain paper.
4. As an arXiv user, I want an accepted published DOI to be recorded as an explicit identity promotion, so that the relationship between the preprint and published work is auditable.
5. As a title-query user, I want loose search to find possible candidates without treating search similarity as identity proof, so that recall does not weaken safety.
6. As an author of a paper whose title contains meaningful symbols, I want C++, C#, Na+, and similar notation preserved during identity comparison, so that semantically distinct papers are not conflated.
7. As a user with incomplete metadata, I want author and year evidence to support a legitimate equivalent manifestation, so that safe enrichment still works when a shared DOI is absent.
8. As a user with weak evidence, I want the pipeline to ask for a stronger identifier or source rather than guess, so that DeepPaperNote remains evidence-first.
9. As a Zotero user with a broken attachment URL, I want valid parent metadata preserved, so that one unusable source does not destroy the Paper Identity.
10. As a Zotero user with several attachments, I want only attachments belonging to the accepted parent item considered, so that unrelated files cannot be fetched.
11. As a user with a valid local PDF, I want that trusted source retained as the Source Manifestation, so that page and figure grounding remain bound to the file actually read.
12. As a user with several valid manifestations, I want source priority to be deterministic and visible, so that acquisition behavior is reproducible.
13. As a user, I want rejected provider observations excluded from both metadata and PDF selection, so that a rejected candidate cannot contaminate the note indirectly.
14. As a user, I want malformed or unavailable optional sources reported as scoped diagnostics, so that recoverable source failures do not become misleading identity failures.
15. As a maintainer, I want one pure adjudication seam, so that adversarial identity cases can be tested without network or filesystem dependencies.
16. As a maintainer, I want every accepted field and source location linked to its provider observation and decision reason, so that regressions can be diagnosed from artifacts.
17. As a maintainer, I want rejected observations recorded in the Identity Repair Trace, so that the system explains why plausible candidates were not used.
18. As a maintainer, I want PDF fetch to be independent of metadata providers, so that acquisition does not reopen identity after acceptance.
19. As a maintainer, I want all direct and pipeline callers to use the same Identity Verification Boundary, so that no legacy shortcut has weaker safety.
20. As a maintainer, I want the existing stage order and artifact audit trail preserved, so that this fix does not create a second workflow.
21. As a note writer, I want only accepted metadata in the synthesis bundle, so that an incorrect abstract, venue, or year cannot influence final prose.
22. As an evaluator, I want identity verdicts and source bindings propagated into downstream artifacts, so that final-note claims can be traced to the manifestation actually read.
23. As a contributor, I want failure classes to distinguish ambiguity, contradiction, provider unavailability, and unsafe sources, so that each failure has an actionable remedy.
24. As a contributor, I want the full existing test suite to remain green, so that the Zotero integration and canonical workflow stay compatible with the rest of DeepPaperNote.

Implementation Decisions

• Keep the existing resolve → metadata collection → identity contract → PDF fetch stage order. Do not add a second public stage or adapter-owned workflow.
• Use the existing identity-contract stage as the single authoritative adjudication seam. It receives the resolution anchor plus separate provider observations and owns admission, repair, rejection, and source binding.
• Represent each observation with its provider, retrieval basis, claimed identifiers and metadata, source locators, parent/attachment relationship where applicable, and scoped diagnostics. Provider adapters only retrieve and normalize observations; they do not decide canonical identity.
• Keep loose normalization for candidate retrieval separate from identity equality. Identity comparison must use Unicode normalization, case normalization, and whitespace normalization while preserving meaningful symbols. A title lookup result is an observation, not an accepted identity.
• Treat shared DOI, arXiv ID, or trusted Zotero relationship as strong equivalence evidence. Competing strong identifiers are a contradiction and must not be resolved by title score.
• When no strong identifier is shared, require a symbol-preserving title match, normalized full leading-author agreement rather than surname-only agreement, and compatible publication year before automatic admission. Supporting abstract or first-page evidence may strengthen or reject the decision but does not make title-only evidence sufficient.
• Perform metadata merge only after an observation has been admitted. Accepted observations may fill missing descriptive fields but may not overwrite non-empty conflicting identity fields. Preserve field-level provenance in the canonical decision.
• Make identity promotion explicit. Adding a DOI or arXiv ID that changes the effective paper ID must appear as an accepted repair/promotion with its supporting evidence; recomputing a paper ID after an incidental merge is not sufficient.
• Upgrade the Canonical Identity Artifact and Identity Repair Trace schemas to carry accepted metadata, ordered bound sources, accepted evidence, rejected observations and reasons, verdict, warnings, failure class, and provenance. Migrate all internal consumers in the same change rather than maintaining a parallel unsafe schema path.
• Bind a source only when it is a trusted explicit input, a Zotero attachment related to the accepted parent, deterministically derived from an accepted strong identifier, or supplied by an observation already accepted as an equivalent Paper Manifestation.
• A raw PDF URL is a locator, not Identity Evidence. A rejected observation's URL must never enter the bound-source list.
• Treat malformed URL parsing, missing attachment files, and download failures as source-scoped diagnostics. Preserve valid parent identity evidence and continue to the next already-bound source when one exists.
• Make PDF fetch transport-only: require an accepted Canonical Identity Artifact, attempt only its ordered bound sources, validate PDF bytes, and report attempted sources. Remove provider queries and metadata enrichment from this stage.
• Remove or close legacy raw-input shortcuts that perform enrichment or source selection without an accepted identity contract. All identity-bearing provider results must route through the single adjudicator.
• Ensure model-facing synthesis reads accepted metadata from the canonical decision and retains Evidence Location Binding to the Source Manifestation actually fetched. Raw observations remain audit evidence, not writing input.
• Keep orchestration and host adapters thin. They may pass artifact locations and configuration but may not redefine identity policy.
• Do not add a provider registry, configurable policy language, class hierarchy, or new dependency. Implement the minimum pure decision logic needed by the existing providers.
• Preserve the five contributor commits already integrated on the maintainer branch; add the identity-safety work as maintainer commits on top of the history-preserving merge.

Testing Decisions

• Prefer one high-level behavioral seam: invoke the real identity-contract boundary with crafted resolution and observation artifacts, then assert the Canonical Identity Artifact and Identity Repair Trace. Tests should observe verdicts, accepted metadata, bound sources, rejections, and failure classes rather than private helper implementation.
• Extend the existing acquisition-artifact contract tests as the primary prior art. Use the existing fetch, Zotero Local API, and pipeline-manifest tests only for boundary wiring that the central contract test cannot prove.
• Parameterize the central contract test over these behaviors:
  • a shared strong identifier accepts the observation's non-conflicting metadata and bound source;
  • a same-title candidate with a conflicting DOI is rejected or fails closed;
  • a missing DOI is not filled from an unsupported same-title observation;
  • a candidate with symbol-different title variants such as C++/C, C#/C, and Na+/Na is not equivalent;
  • a strict title plus full leading-author and compatible-year match may be accepted as an equivalent manifestation;
  • a rejected observation's PDF URL is absent from bound sources;
  • explicit identity promotion records the accepted correction and evidence;
  • insufficient or competing evidence produces the correct fail-closed class and user-facing summary.
• Extend Zotero boundary tests to prove that malformed parent and attachment URLs preserve parent identity, record zotero_unsafe_file_url, and do not expose the unsafe locator.
• Extend fetch tests to prove that an accepted bound source can be downloaded while metadata enrichment is forced to fail if called. This demonstrates that fetch performs no enrichment.
• Extend the pipeline manifest test to prove that an ambiguous or failed identity stops before fetch and that an accepted identity's provenance, bound source, and accepted metadata survive into the source manifest and synthesis bundle.
• Add a caller audit assertion or equivalent focused check showing that acquisition-stage consumers cannot invoke the legacy enrichment path after canonical identity acceptance.
• Use deterministic fake provider observations and fake download bytes. Network availability must not affect identity-policy tests.
• Run the full repository suite with the repository's required Python runtime after focused tests pass.
• The change is complete only when all identity-bearing provider results pass through the adjudicator, rejected candidates cannot contribute fields or locations, fetch performs no enrichment, downstream writing input uses accepted metadata, all adversarial regressions pass, and the full suite remains green.

Out of Scope

• A generic provider plugin framework, provider registry, or configurable identity-policy language.
• New metadata providers or changes to provider ranking unrelated to identity admission.
• Redesigning Source Corpus extraction, evidence extraction, figure planning, note drafting, linting, Final Quality Review, or Formal Save.
• Model-led identity adjudication or network-dependent semantic matching.
• Proving PDF-content equivalence beyond the existing acquisition and Source Manifestation contract.
• Fetch-time identity repair or post-fetch metadata enrichment.
• A new public pseudo-command or a second canonical workflow.
• Unrelated README, adapter, packaging, or formatting refactors.
• Squashing or rewriting the integrated contributor commits.

Further Notes

• The implementation target is the maintainer integration branch integrate/pr-13-identity-safety, currently based on the history-preserving merge of PR feat: add native Zotero Local API resolution #13.
• The existing identity-contract stage already has the right pipeline position, fail-closed exit behavior, audit artifacts, and downstream gate. The change should deepen that seam rather than add another one.
• The main regression risk is leaving one direct enrich_metadata, title-search merge, or standalone fetch path that bypasses adjudication. Before completion, audit every caller and remove or reroute all identity-bearing shortcuts.
• This work intentionally accepts a stricter user experience: when independent evidence is insufficient, DeepPaperNote should request a DOI, arXiv ID, Zotero key, or trusted PDF instead of guessing.