Should we explore DiskANN for aKNN vector search?

[mikemccand]

Description

I came across this compelling sounding JVector project which looks to have awesome QPS performance.

It uses DiskANN instead of HNSW (what Lucene uses now).

Maybe we should explore another aKNN Codec implementation using this? It'd also be a good test of the Codec pluggability of our KNN implementation.

[benwtrent]

I do think Lucene's read-only segment based architecture leads itself to support quantization (required for DiskANN).

It would be an interesting experiment to see how indexing times, segment merges, etc. are all handled with DiskANN.

I wonder how much the speed difference is:

• Vectors being out of memory (and if they used PQ for diskann, if they did, we should test PQ with HNSW).
• Not merging to a single segment and searching multiple segments serially.

[robertvanwinkle1138]

@benwtrent
For merges there is "FreshDiskANN: A Fast and Accurate Graph-Based
ANN Index for Streaming Similarity Search"
https://arxiv.org/pdf/2105.09613.pdf

DiskANN is known to be slower at indexing than HNSW and the blog post does not compare single threaded index times with Lucene.

[benwtrent]

DiskANN is known to be slower at indexing than HNSW and the blog post does not compare single threaded index times with Lucene.

@robertvanwinkle1138 this is just one of my concerns.

Additionally, I think we would still have the "segment merging problem". I do think we can get HNSW merges down by keeping connections between graphs, I just haven't had cycles myself to dig into that. There is a Lucene issue around making HNSW merges faster somewhere...

#12440

[jmazanec15]

A hybrid disk-memory algorithm would have very strong benefits. I did run a few tests recently that confirmed HNSW does not function very well when memory gets constrained (which I think everyone already knew).

I wonder though, instead of DiskANN, what about a partitioning based approach such as SPANN? I think a partitioning based approach for Lucene would make merging, updating, filtering and indexing a lot easier. Also, it seems it would have better disk-access patterns. In the paper, they do show that in a memory constrained environment, they were able to outperfrom DiskANN.

I guess the tradeoff might be that partitioning based approaches would struggle to achieve really low latency search when in memory compared to graph-based approaches. Additionally, partitioning approaches would require a potentially expensive "training" or "preprocessing" step such as k-Means and performance could degrade if data distribution drifts and the models are not updated. But, if PQ is implemented, the same considerations would need to be taken as well.

[benwtrent]

@jmazanec15 I agree that SPANN seems more attractive. I would argue though we don't need to do clustering (in the paper they do clustering, but with minimal effectiveness), but could maybe get away with randomly selecting a subset of vectors per segment.

[robertvanwinkle1138]

The SPANN paper does not address efficient filtered queries. For example, Lucene's HNSW calculates the similarity score for every record, regardless of the record matching the filter.

Filtered − DiskANN [1] describes a solution for efficient filtered queries.

QDrant has a filter solution however the methodology described in their blog is opaque.

1. https://dl.acm.org/doi/pdf/10.1145/3543507.3583552

As Approximate Nearest Neighbor Search (ANNS)-based dense retrieval becomes ubiquitous for search and recommendation scenarios, efciently answering fltered ANNS queries has become a critical requirement. Filtered ANNS queries ask for the nearest neighbors of a query’s embedding from the points in the index that match the query’s labels such as date, price range, language. There has been little prior work on algorithms that use label metadata associated with vector data to build efcient indices for fltered ANNS queries. Consequently, current indices have high search latency or low recall which is not practical in interactive web-scenarios. We present two algorithms with native support for faster and more accurate fltered ANNS queries: one with streaming support, and another based on batch construction. Central to our algorithms is the construction of a graph-structured index which forms connections not only based on the geometry of the vector data, but also the associated label set. On real-world data with natural labels, both algorithms are an order of magnitude or more efcient for fltered queries than the current state of the art algorithms. The generated indices also be queried from an SSD and support thousands of queries per second at over 90% recall@10.

[benwtrent]

QDrant has a filter solution however the methodology described in their blog is opaque.

QDrant's HNSW filter solution is the exact same as Lucene's. You can look at the code, they don't filter candidate exploration but filer result collection.

You are correct that filtering with SPANN would be different. Though I am not sure its intractable.

It is possible that the candidate postings (gathered via HNSW) don't contain ANY filtered docs. This would require gathering more candidate postings.

But I think we can do that before scoring. So, as candidate posting lists are gathered, ensure they have some candidates.

The SPANN repository supports filtering, and its OSS, so we could always just read what they did.

[robertvanwinkle1138]

QDrant's HNSW filter solution is the exact same as Lucene's

Interesting thanks.

as candidate posting lists are gathered, ensure they have some candidates

Couldn't that be done with the current Lucene HNSW implementation?

The SPANN repository supports filtering, and its OSS, so we could always just read what they did.

This search with filter method seems to throw an error.

https://github.com/microsoft/SPTAG/blob/main/AnnService/src/Core/SPANN/SPANNIndex.cpp#L262

[benwtrent]

This search with filter method seems to throw an error.

LOL, I thought it was supported, I must have read a github issue and made an assumption.

Couldn't that be done with the current Lucene HNSW implementation?

I think so. The tricky thing is making sure enough matching postings are gathered to give an accurate number for recall. Having to go back to the HNSW graph to get more postings list could get expensive if we keep getting to few matches.

[mikemccand]

SPANN is another option?

https://www.researchgate.net/publication/356282356_SPANN_Highly-efficient_Billion-scale_Approximate_Nearest_Neighbor_Search

[mikemccand]

(listening to @jbellis talk at Community over Code).