Benchmarks: add MedCPT (PubMed) and bge-small recall and agreement cells #567

Following up on #562: a proposal for adding the two bunched corpora to benchmarks/suite/, so the api.md MedCPT row and a bge-small row can be regenerated from the suite and checked on every release. Scripts for everything below already exist in https://github.com/nickjlamb/turbovec-biomed-bench. The PR would adapt them to the suite's conventions.

Corpora

Both are below the suite's existing sizes and use the same split as recall_*.py: seed-42 permutation, then 100K database and 1K queries.

Loader

Two new targets in benchmarks/download_data.py, following the existing pattern: write to ~/data/py-turboquant/<name>.npy via a temp file and an atomic rename.

• medcpt-768: reads each chunk's .npy header and checks the dtype and shape, then range-reads the two row slices. It also saves the matching PMIDs as medcpt-768.pmids.json, so any query can be traced back to an article.
• bge-small-pubmed-384: depends on the data question below.

Each target checks a sha256 of its output, so the corpus can't drift if NCBI republishes a chunk.

Cells

• Recall: recall_medcpt_{2,4}bit.py and recall_bgepubmed_{2,4}bit.py, the same as the existing recall cells: TQ, TQ+ (calibrate() on 1,024 rows) and FAISS IndexPQ at a matched bit rate, recall@1@k at k=64, with the same JSON schema in benchmarks/results/.
• Agreement: agreement_{corpus}_{2,4}bit.py, staged search vs whole-index scan, to regenerate the api.md agreement rows:
  • Each mode runs in its own subprocess, because TURBOVEC_{2,4}BIT_PLANES is read once per process.
  • k = 1, 10 and 100, over 3 seeds × 1,000 queries.
  • It reports identical id sets, ids shared, recall k@k (staged and full) and whether the true nearest neighbour is in the top k, matching the api.md columns.
  • I'd include openai-1536 too, so the suite reproduces every public agreement row.

Two details that need deciding

1. Ties in the ground truth. The MedCPT slice has 605 exact-duplicate rows (mostly errata notices and non-English records embedded from empty text; see Precomputed PubMed embeddings: about 0.2–0.3% of articles are embeddings of an empty input ncbi/MedCPT#8). They give 1.0–1.4% of queries an exact tie for top-1, so the suite's argmax ground truth scores some correct answers as misses. In Staged search matches the whole-index scan on ~83% of queries at k=10 on PubMed (MedCPT) embeddings #562 I counted a hit when the returned id's exact score was within 1e-6 of the best. Should the new cells do that, or should the loader drop duplicate rows instead? The existing corpora would be unaffected either way: DBpedia has none.
2. Silent passes on the wrong hardware. The staged path needs ≥32,768 vectors and, on x86, AVX-512 VBMI and VNNI. Without them both modes run the whole-index scan and agreement is trivially 100%. I hit this today: re-running on 1.1.2 from PyPI, my VM had moved to a CPU without VBMI and every cell came back 100%. I'd have the agreement cells record the CPU flags and fail loudly when the staged path can't run, unless there's a better way to ask the index which path it used.

Questions

1. bge-small data: should the loader regenerate it (sentence-transformers plus ~3.4 GB of NCBI text JSON, about 2 hours on 2 CPU cores), or download a pinned .npy (~155 MB) published once with its sha256, keeping the embedding script in-tree for provenance? I'd lean towards the pinned file.
2. Ties: tie-aware hits or deduplicated rows (above)?
3. Placement: do you want the agreement cells in suite/, or feeding your release gate somewhere else?
4. Diagrams: a recall panel in create_diagrams.py for the new corpora, or JSON only?

Licensing to confirm before the PR: bge-small-en-v1.5 is MIT. I'll check the terms on NCBI's MedCPT embeddings and note them in the loader.