perf(sync): incremental journal/conference fetch via Railway archive

[kishormorol]

Problem

Journal-sync and conference-sync re-download the entire corpus every run. The latest journal-sync run took ~2.6 hrs, with 157.9 min (99%) spent in the OpenAlex journal fetch — re-pulling the full 2022→now window across all 25 journals (NatComms alone is ~44k works). The fetch never consulted what was already stored, so dedup silently discarded ~99% of re-downloaded papers.

Fix

Make every heavy sync fetch incremental, backed by the complete uncapped archive that already lives in Railway Postgres. (The committed *_db.json files are capped top-N working sets, not complete archives — and a full conferences_db.json would exceed GitHub's 100 MB file limit, so it can't live in git.)

• Journals — OpenAlex from_created_date filter pulls only works indexed since the last run. Watermark in site/data/journal_sync_state.json (minus 7-day lookback).
• Conferences — proceedings are immutable per (venue, year); settled past-year blocks are skipped via skip_keys threaded into every conference connector's fetch_all (openreview/pmlr/cvf/s2/acl). Current calendar year is always refetched.
• OpenAlex concept bulk-fetch (ML/NLP/CV/IR) — was re-paginating entire concepts (100k+ works) uncapped each run; now incremental too.

Safety

• Incremental/skip engages only when Railway loads successfully (railway_store.load / _load_complete_archive). If Railway is down → full re-fetch, never silent data loss.
• A venue-name mismatch can only cause a missed skip (full fetch), never a wrong skip.

Rollout

1. Merge.
2. One seeding run does a full backfill (last slow run) to populate Railway + write the watermark.
3. Every run after → incremental. Hours → minutes.

Tests

142 tests pass locally (2 pre-existing env-only collection errors — libopenblas, tinydb — unrelated). Verified the incremental filters emit from_created_date, full mode omits it, settled-keys logic, and the Railway dict → Paper round-trip.

🤖 Generated with Claude Code