Building email logical threader and de-duplicator with haystack

[ayushin]

Hi guys,

I am completely new to Haystack and I wonder if this is the right tool to do email logical threading.

What I need is to feed a corpus of emails and build logical threads (graphs) between senders and receivers, similar to what some of the law e-discovery tools do, for example https://blog.ayfie.com/ayfies-power-features-part-2-brand-new-email-threading

So basically all emails need to be broken down into fragments (with some near matches) that are later organised into graphs to reconstruct the entire corpus in threads even if some emails are missing or the subject of the thread was changed or the headers were not kept intact.

Before I invest time into learning haystack, is this something that can be easily built with Haystack pipelines?

Or should I look into building something just on top of a vector database instead?

[masci]

Unfortunately Haystack doesn't provide a turnkey solution for your use case, but if the alternative is building something from scratch on top of a vector database, Haystack can still do the heavy lifting for you: preprocessing the email messages, dealing with metadata, comparing different similarity models are a few tasks that you have to deal with eventually, and that Haystack can do easily.