De-Dup Engine

In our approach for the dedup engine, we will be using dedupe module that uses machine learning to perform matching, deduplication and entity resolution quickly on data.

Below is the flow of project:

The API we are going to build will provide a variety of customizable options, so that duplicates can be found as per the requirements and more of flexibility.