People search eval

data/people-search-eval/.env.example

@@ -0,0 +1,22 @@
+# Copy from people-search-eval/.env or set here
+NYNE_API_KEY=
+NYNE_API_SECRET=
+PDL_API_KEY=
+BRAINTRUST_API_KEY=
+BRAINTRUST_PROJECT=people-data-provider-evals
+BRAINTRUST_LEGACY_IDS=true
+BRAINTRUST_DATASET=full_provider_benchmark
+BRAINTRUST_OVERALL_SCORER=people-judge-overall
+BRAINTRUST_PERSONA_SCORER=people-judge-persona
+NYNE_INSIGHTS=0
+NYNE_PROFILE_SCORING=0
+NYNE_PROBABILITY_SCORE=0
+NYNE_SEARCH_LIMIT=5
+NYNE_EVAL_CONCURRENCY=1
+PDL_SEARCH_SIZE=5
+EXA_API_KEY=
+EXA_NUM_RESULTS=5
+EXA_SEARCH_TYPE=auto
+EXA_QUERY_FIELD=auto
+EXA_INCLUDE_HIGHLIGHTS=1
+EXA_EVAL_CONCURRENCY=1

data/people-search-eval/README.md

@@ -0,0 +1,48 @@
+# People Search Eval
+
+240-query people-search benchmark (6 personas × 40) for Braintrust.
+
+## Layout
+
+```
+data/people-search-eval/
+├── dataset/                 # query set
+│   └── full_provider_benchmark.json
+├── scorers/                 # build / publish judges + deterministic scorers
+│   ├── prompts/
+│   ├── field_fill_scorer.py
+│   ├── persona_field_fill_scorer.py
+│   └── publish_scorers.py
+└── eval/                    # run evals
+    ├── run_nyne.py
+    ├── run_pdl.py
+    ├── run_exa.py
+    └── lib/                 # shared runner helpers
+```
+
+See [SCORERS.md](SCORERS.md) for scorer design.
+
+## Setup
+
+```bash
+cd data/people-search-eval
+python3 -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+
+# Provider clients (Nyne / PDL / Exa)
+pip install -e ../../../people-search-eval
+# or: export PEOPLE_SEARCH_EVAL_SRC=/path/to/people-search-eval/src
+
+cp .env.example .env   # set API keys
+python scorers/publish_scorers.py
+```
+
+## Run
+
+```bash
+python eval/run_nyne.py --local-dataset --limit 3 --no-send-logs --low-credits
+python eval/run_pdl.py --local-dataset --limit 3 --no-send-logs
+python eval/run_exa.py --local-dataset --limit 3 --no-send-logs
+
+python eval/run_nyne.py --project people-data-provider-evals --dataset full_provider_benchmark --low-credits
+```

data/people-search-eval/SCORERS.md

@@ -0,0 +1,53 @@
+# SCORERS.md — people-search eval
+
+Dataset: `dataset/full_provider_benchmark.json` (240 rows, 6 personas × 40 queries).
+
+## Scorer layers
+
+### Layer 1 — Deterministic (Python, free)
+
+| Scorer | Name | Range | What it measures |
+|--------|------|-------|------------------|
+| `has_people` | person_count | 0 or 1 | At least one person returned |
+| `field_fill` | field_fill | 0–1 | Mean fill ratio across 11 universal fields |
+| `persona_field_fill` | persona_field_fill | 0–1 | Same fields, weighted by buyer persona |
+
+**Hard rules:** `error` or `person_count=0` → deterministic scores are **0**.
+
+### Layer 2 — LLM judges (Braintrust UI)
+
+| Slug | Audience |
+|------|----------|
+| `people-judge-overall` | Cross-persona quality |
+| `people-judge-persona` | Persona-switched — one prompt that applies the rubric for the row's `metadata.persona` |
+
+Choice scores (all judges):
+
+| Label | Score |
+|-------|-------|
+| High Value | 1.0 |
+| Useful | 0.7 |
+| Low Value | 0.3 |
+| Failed | 0.0 |
+
+**Calibration rule:** named-person `enrichment` with zero results → **Failed (0.0)**, not partial credit.
+
+Prompt sources: `scorers/prompts/*.md`
+
+## Setup
+
+```bash
+cd data/people-search-eval
+pip install -r requirements.txt
+pip install -e ../../../people-search-eval
+cp .env.example .env
+python scorers/publish_scorers.py
+```
+
+## Run evals
+
+```bash
+python eval/run_nyne.py --local-dataset --limit 3 --no-send-logs --low-credits
+python eval/run_pdl.py --local-dataset --limit 3 --no-send-logs
+python eval/run_exa.py --local-dataset --limit 3 --no-send-logs
+```