Simplify the encoding and deduplication process#268

datamik wants to merge 3 commits intodevelopunicef/hope-dedup-engine:developfrom

improve-deduplication-processunicef/hope-dedup-engine:improve-deduplication-processCopy head branch name to clipboard