Use a word list for lookup

[monWork]

Hi,
I have a list of words that i want to use to filter out the sentences in the document and use the filtered sentences only for training and inferencing.
I did look into spacy lookup as option but need help with the best way i can do that and make it efficient and how i can pass that list into my training config file ?

[ljvmiranda921]

Hi @monWork , I suggest looking into PhraseMatcher for your task. Here's a good example on how you can do rule-based matching given a list of terms.

[monWork]

Thanks @ljvmiranda921 , do you have example which I look at integrate this as an component in my training config? Or it would be more of using a custom code for component and I do phrase matching in there.

[ljvmiranda921]

Hi @monWork , ideally you want to do the filtering first before saving everything into the spaCy file format. Using a custom component might be a neat option, but it might be better to think of the filtering step as a preprocessing one.