How remove similar items in big datasets?

[info2000]

There's some idea to find and remove similar items in datasets?
I want to my eval datasets be the most diversity and light possible, finding and removing similar items, without running a loop of each element against the others elements

Thanks

[polm]

Finding similar documents is a whole field of research so it's hard to give a brief summary, but you could use one of the spaCy pipelines with word vectors and a nearest neighbor search library like Annoy as a starting point - it should be much faster than a nested for loop. You would have to define yourself how similar an item has to be before it needs to be removed.

[info2000]

Thanks, using Annoy works really faster.
if someone is on this situation, I filter all the samples with distance below 0.15,
A dataset with 1 millon examples takes minutes vs hours doing similarity loop over items loops