references

A personal academic literature collection for research in industrial-organizational psychology, psychometrics, and applied statistics. The repository tracks the search infrastructure built around the collection; the PDFs, .txt files, and other copyrighted documents are excluded from version control (see .gitignore) to comply with publisher terms of service.

What is committed

File	Purpose
build_catalog.py	Extracts metadata and full text from PDFs; writes search artifacts
references_catalog.parquet	Structured metadata catalog (2,253 papers, 292 subdirectories, 1990-2026)
CLAUDE.md	Claude Code configuration for this directory
.gitignore	Excludes PDFs, .txt, .docx, .ppt, and other copyrighted content

What is not committed

PDFs, extracted .txt files, Word documents, and slide decks are gitignored. These files are available locally for personal research use under standard academic licensing terms; redistribution would violate publisher agreements.

Setup

# Python version is pinned via pyenv; direnv loads the venv automatically
pyenv install            # installs version from .python-version
python3 -m venv .venv
source .venv/bin/activate
pip install pymupdf pyarrow

build_catalog.py

Walks the references directory recursively, extracts metadata from each PDF using pymupdf, and writes references_catalog.parquet for fast DuckDB queries.

python3 build_catalog.py [--force] [--subdir SUBDIR] [--verbose]
                         [--extract-text] [--text-only]

  --force          Re-extract all PDFs, ignoring cached entries
  --subdir SUBDIR  Process only a specific subdirectory (e.g. stat-methods/glmm)
  --verbose        Print per-file status
  --extract-text   Also write full-text .txt files alongside each PDF
  --text-only      Write .txt files only; skip catalog build

Extraction is incremental: files are skipped when the cached entry is newer than the PDF. --force bypasses this.

Search workflow

The catalog supports a two-stage retrieval pattern.

Stage 1 — abstract search (DuckDB): fast, returns papers that foreground a concept in their abstract.

duckdb -c "
SELECT file_path, title, authors, year
FROM read_parquet('references_catalog.parquet')
WHERE abstract ILIKE '%random slope%'
  AND subdirectory LIKE 'stat-methods%'
ORDER BY year DESC;
"

Stage 2 — full-text search (ripgrep): catches papers that apply a method without naming it in the abstract. Requires --extract-text to have been run.

rg -l "random slope" stat-methods/glmm/

Abstract search alone misses roughly 20% of the corpus (papers with no extractable abstract). Full-text search provides 7-10x higher recall on method-specific terms in core research subdirectories.

/litreview skill

This project is the data backend for the /litreview Claude Code skill defined in datasci-iopsy/.dotfiles at claude/skills/litreview/SKILL.md.

The skill provides a structured five-step retrieval workflow:

1. Catalog query (DuckDB) — filters by subdirectory, title, and abstract keywords; targets 8-15 candidates
2. Abstract triage (in context) — ranks candidates without reading any PDFs; selects 3-5 for deep read
3. Full-text search (ripgrep) — optional; uses the extracted .txt files to confirm method or term presence in the paper body when abstracts are insufficient
4. Deep read (Read tool) — reads only the selected 3-5 PDFs; hard limit per pass
5. Synthesis — produces thematic synthesis across papers; never a serial summary list

To invoke it from any Claude Code session in this directory:

/litreview glmm
/litreview "power analysis longitudinal"
/litreview burnout-engagement

The skill checks for references_catalog.parquet at startup. If the catalog is missing, it halts and prompts you to run python3 build_catalog.py first. Full-text search (Step 3) requires the .txt files produced by --extract-text; the skill detects their presence automatically and skips Step 3 if they are absent.

Catalog schema

Column	Type	Notes
file_path	string	Relative path from repo root
subdirectory	string	Immediate parent directory
filename	string	PDF filename
title	string	Extracted from metadata or page text
authors	string	Extracted from metadata or page text
year	int64	Publication year
abstract	string	Up to 500 words
page_count	int64
quality_score	float64	Fraction of title/authors/year/abstract present (0-1)
pdf_warnings	string	MuPDF warnings, if any
extracted_at	string	UTC ISO timestamp

For interactive exploration of the catalog outside the terminal, Parquet Explorer is a good option. Claude Code users may also find it useful to invoke the /duckdb skill for query-assisted analysis directly in the chat session.