Describe the bug
When we supply to cagraIndexBuild (or the equivalent C++ functions) a dataset that resides in device memory (e.g. because it was already copied, e.g. to remove strides as a workaround for #1455, or is the result of other GPU computations like quantization), a new copy of the dataset is created in device memory. This means we need (at least) twice the dataset memory on the device, meaning cagraIndexBuild is likely to fail for bigger datasets.
Steps/Code to reproduce bug
On a X GB GPU, supply to cagraIndexBuild a DLManagedTensor with dataset type kDLCUDA (dataset address on device memory) and size X * 0.6 GB (a bit more than half the ram, e.g. a 13GB dataset on a 24GB GPU). cagraIndexBuild fails with a memory error.
Expected behavior
cagraIndexBuild should provide help/guidance/a way to avoid that:
- a way of not copying accepting the performance hit could be fine
- instructions on how to shape the data (pre-stridden/padded data) so we can avoid the copy and take a "fast path"
Any way to avoid the issue and allow more data to be fitted and processed on the GPU
Additional context
Another issue that was recently brought to light is the fact that nn-descent wants to copy the whole dataset into float16.
That is a separate issue above and beyond the strided/padded dataset issue, but it's another area of concern around usage of GPU memory/multiple dataset copies.
Describe the bug
When we supply to
cagraIndexBuild(or the equivalent C++ functions) a dataset that resides in device memory (e.g. because it was already copied, e.g. to remove strides as a workaround for #1455, or is the result of other GPU computations like quantization), a new copy of the dataset is created in device memory. This means we need (at least) twice the dataset memory on the device, meaningcagraIndexBuildis likely to fail for bigger datasets.Steps/Code to reproduce bug
On a X GB GPU, supply to
cagraIndexBuilda DLManagedTensor with dataset type kDLCUDA (dataset address on device memory) and size X * 0.6 GB (a bit more than half the ram, e.g. a 13GB dataset on a 24GB GPU).cagraIndexBuildfails with a memory error.Expected behavior
cagraIndexBuildshould provide help/guidance/a way to avoid that:Any way to avoid the issue and allow more data to be fitted and processed on the GPU
Additional context
Another issue that was recently brought to light is the fact that nn-descent wants to copy the whole dataset into float16.
That is a separate issue above and beyond the strided/padded dataset issue, but it's another area of concern around usage of GPU memory/multiple dataset copies.