Script to generate billion scale synthetic data

[singhmanas1]

This script generates billion-scale synthetic data. Following is the procedure-
Dataset Generation-

1. Generate centroids using k-means over randomly generated 1M vectors.
2. Create a Gaussion blob for each centroid using varying standard deviation.
   Ground Truth Generation-
3. Randomly sample query vectors from dataset (from each cluster) -- remove these IDs from the dataset.
4. Use cuvs brute force to identify 4 closest centroids
5. Do a brute force search over the 4 clusters to identify top-k ground truth.

User Configurations-

1. Total vectors
2. Total vectors per cluster
3. Vector dim and dtype
4. Top-k (for ground truth)

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[singhmanas1]

Time to generate 10B vectors with 1 L4 (AWS g6.16xlarge)