Improvements to the Faiss codec

[kaivalnp]

Description

Lucene added a Faiss-based vector search format in #14178, opening this issue to track possible enhancements:

1. Allow Faiss to use SIMD instructions (more context here)
2. Byte vector support (more context here)
3. Implement batched indexing (more context here)
4. Mmap-ed IO with disk-based indexes
   • Today, the Faiss codec does not make any assumptions about the type of the Lucene index, and consequently uses abstract methods to read / write bytes from Faiss indexes
   • Faiss recently added mmap support for some indexes, making it possible to read indexes without loading it entirely into RAM -- and we should make use of this functionality wherever possible / desired
5. GPU support
   • Faiss has support for GPU-based indexes using CUDA or ROCm -- but we do not use it from Lucene today. This may be a shorter path to Support Vector Search on GPU with cuVS #14243
   • One reason is that GPU-based searches are beneficial when running queries in parallel, whereas the Faiss codec today runs queries one-at-a-time -- i.e. we may need a "batching" layer in the codec
6. Expose a safe schema for Faiss indexes
   • Today, the Faiss codec can be configured using an index factory string
   • While this provides flexibility to the user, it is also error-prone (syntax errors, using incompatible features, etc). It may be more prudent to expose a safe schema from Lucene and build Faiss indexes programmatically instead of relying on the index factory
   • This has subtle benefits, because Lucene now "knows" the type of Faiss index used -- for example decisions like not requiring a separate copy of raw vectors if the Faiss index is "lossless", or query-level configuration (setting parameters like efSearch, nprobe, etc per-query)
   • One caveat is having to map many of Faiss' features in Lucene / Java as an ongoing effort
7. Support for parent-join queries or timeouts
8. Re-using information from previous indexes at merge-time
   • Today, we simply create a new Faiss index from scratch during merge -- but it may be possible to re-use previous indexes to speed up merges

Please feel free to add enhancements I may have missed, or link issues / PRs

[mikemccand]

• Faiss recently added mmap support for some indexes, making it possible to read indexes without loading it entirely into RAM -- and we should make use of this functionality wherever possible / desired

Cool! Zero copy readers too. I had thought Faiss was RAM heavy, yielding a nice curve from jVector Codec (RAM light), Lucene's default HNSW (nice balance of carefully choosing what should be RAM vs disk), and Faiss (RAM heavy). But it sounds like Faiss is sometimes disk friendly -- these recent mmap improvements, and at least also IVF Flat.

1. Allow Faiss to use SIMD instructions (more context here)

Wow, that means it's impressive that Faiss shows such good performance in luceneutil's knnPerfTest.py when it's using ordinary non-vectorized CPU instructions.