Large scale issue

[seanyang38]

Hi, thank you for the implementation. It's very useful.

However, I am running into problems when using large scale and large dimensionality data. For example, when I change the data points to 10,000 for query, and 800,000 for reference, the knn indices results are either all zeros or random numbers that are larger than 800,000. I attached a screenshot of the console for the experiment. I am wondering what the reasons are for this, and if there's any way to solve this. Thank you in advance.

[zjhthu]

I have met the same problem, have you solved this problem? @seanyang38

[RuibinMa]

Same problem. Maybe because of non-enough memory.

[bigrobinson]

I've had similar problems with the predecessor to this code. There's a good chance you'll find the culprit is this allocation of device memory in knn_cuda.cu in the knn_cuda fcn:

cudaMalloc((void **) &dist_dev, query_width * ref_width * size_of_float);

It is allocating an array (for an adjacency matrix I do believe) of dimension query_width * ref_width. If you have 10,000 query points and 800,000 reference points (3D), you have 10^4 * 8x10^5 * 3 * 4 bytes of data if you're single precision. That's approaching 100GB. My guess is your device has far less memory.