serialization of indices for faiss and voyager backends for large scale use

[apoorvalal]

Persist index by writing to backend-specific binary file. Lets you create an index once and on subsequent runs only incur the lookup cost instead of creating the distance metric from scratch. Can Also be updated with new units.

demo

[Copilot]

Pull Request Overview

This PR implements index persistence for FAISS and Voyager backends, enabling large-scale production use by allowing users to build indices once and reuse them across sessions without incurring the full index construction cost.

• Added save() and load() methods to FAISS and Voyager backend classes with JSON metadata sidecars
• Added save_indices(), load_indices(), and estimate_with_preloaded() methods to the Matching class
• Bias correction models are now persisted using pickle alongside index files

Reviewed Changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
fastmatch/knn_faiss.py	Added save/load functionality with metadata and covariance matrix persistence
fastmatch/knn_voyager.py	Added save/load functionality with metadata and space type persistence
fastmatch/fastmatch.py	Modified fit() to store fitted models and added save/load/estimate methods for persistent indices
tests/test_persistence.py	Added roundtrip tests for FAISS, Voyager backends and ATE estimand
examples/persistent_indices.ipynb	Added demo notebook showing index persistence workflow
PERSISTENCE_SPEC.md	Added specification document for the persistence feature

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.