Repository navigation
[Performance] Prefetch for on-disk inverted list #5142
Replies: 3 comments
|
@pseudo-xqr how many vectors at what dimension do you have? |
|
A data point that agrees with @mnorris11: in our case prefetching only paid off once the data stopped fitting in RAM. Setup: IVF-PQ, about 36M vectors (512-d), 16,384 lists, nprobe 1,536, the index read with Warm, a 30-vector query took 0.65 s. Cold it took 18.3 s, because each of those scattered 512-byte rerank rows was its own page fault. More prefetch threads wouldn't have helped there. What fixed it was one batched OS call per query for exactly the pages we were about to read, issued after the IVF step when the row ids are known ( So for hotpotqa, which I'd guess fits in the page cache after the first pass, any prefetch is probably pure overhead, which is what your table shows. It starts to matter when the working set is far larger than memory and the reads are scattered. Write-up with the code, if useful: https://dev.to/laserygd/how-i-built-a-reverse-video-search-engine-on-one-home-pc-o61 (disclosure: it's the index behind a search site I run). |
Uh oh!
There was an error while loading. Please reload this page.
Summary
Enabling prefetch with the default number of prefetch threads (32) for onDiskInvertedLists seems to degrade overall performance.
Platform
OS: Linux
Faiss version: v1.13.2-7-gc9bab48d1
Running on: CPU with 64 cores
Interface: C++ (Python should work as well)
Dataset used: hotpotqa
Reproduction instructions
I prebuilt the vector database on hotpotqa dataset with all-MiniLM-L6-v2 model, and save the trained vector database as the onDiskInvertedLists. The core code of running search is attached below:
In
index_ivf->searchfunction, if I comment out the function callprefetch_listsin thesub_search_func, the overall search latency is lower than without commenting out the function call.Another thing that I have tried is to reduce the number of prefetch threads, but the performance is still worse than without prefetching. The experiments that
search_timecaptured byivf_statsIs this performance trend as expected to see performance degradation with the default 32 prefetch threads? Under what configuration does using 32 prefetch threads be performance-wise beneficial?
All reactions