[Exact Substring Deduplication] Analysis

[ChenghaoMou]

Near deduplication #7 only operates on file level. It is also possible for a file to be

1. a substring of another file, while the minhash/simhash fingerprints being wildly different
2. composed of multiple snippets from different sources

Do we do something about them, knowing they contains large chunks of repeated snippets?

[lvwerra]

How hard would it be to do some analysis of how often this is the case maybe on a subset of data?