Update cagra config yaml for cuvs_bench#1822
Conversation
tarang-jain
left a comment
There was a problem hiding this comment.
LGTM except one comment.
| intermediate_graph_degree: [32, 64, 96, 128] | ||
| graph_build_algo: ["NN_DESCENT"] | ||
| graph_degree: [32, 64, 96] | ||
| intermediate_graph_degree: [128] |
There was a problem hiding this comment.
Dont we want intermediate_graph_degree 64 when the graph degree is 32? @bkarsin had also raised an issue in the past wherein CAGRA can give poor recall when the intermediate graph degree is quite large in comparison to the graph degree.
There was a problem hiding this comment.
Ideally yes, intermediate_graph_degree = 2 * graph_degree would be sufficient. Our benchmark framework does not adjust this automatically. I would not expect recall drop by using larger intermediate degree. Do we have any detail on that that issue?
|
/ok to test e06e1ee |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
📝 WalkthroughSummary by CodeRabbit
WalkthroughThis PR updates hyperparameter sweep configurations for CAGRA and multi-GPU CAGRA (MG_CAGRA) benchmark algorithms. The primary changes include switching the graph build algorithm from ChangesBenchmark CAGRA configuration parameter updates
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This PR updates the CAGRA config options for the parameter sweep used by cuvs-bench.
intermediate_graph_degree: in practice we often useintermediate_graph_degree = graph_degree * 1.5. We pick a fixed value that works works with all of thegraph_degreeoptions we want to test.max_iterations: the default 0 value auto selects the iterations based on itopk. I have added value16additionally, which is a low number of iterations, and it can be useful to reach large throughput in recall < 0.9 region.search_width: we often reach around 99% recall with width 8, therefore I propose to remove higher values from the base scangraph_build_algo: ideally we wound want the algorithm automatically selected. Currently the benchmark framework would select the iterative method by default, which does not work with memory mapped data. The NN descent method can run out of memory for 10M scale dataset. Therefore I propose to fixIVF_PQ, until we enable a reliable default algorithm selection.