Parsing Priority

[HadhemiDS]

I'm trying to parse full street names into street name, street type and street direction using Spacy NER. My training data contains only the cases where the street direction is either E (for east), S (for south), N (for north), or w (for West). I want to integrate other keywords like east, north, south, south-west, etc. To do that spacy uses the entity ruler matcher json file where we can integrate the label as street direction and pattern like this {"label":"streetdirection","pattern":[{"LOWER":{"IN":["east","north","northeast",.. etc. When the model learns from the training data it will not consider those patterns. To consider those patterns there is a flag in the config file that should be set to true overwrite_ents = true. The issue with this is that it overwrites what the model learns from the data and it parses blindly each keyword from the list as it is a street direction. An example of this is The North Street. The north is a street name but when we use the flag to true it will be detected as street direction. How to fix this? Please

[svlandeg]

It sounds to me like ideally, you'd have a NER model that is able to pick up on street names in general - that means training it with sufficiently diverse examples. If you want to combine this NER model with an EntityRuler, I would personally use the EntityRuler as a fall-back mechanism to increase recall - i.e. not have it overwrite the NER predictions but add cases that are most likely correct and can be detected with simple rules/phrase matching.

The details always depend on the exact data & challenge you're addressing, but it'll be worth playing around with the order & settings of these to see what works best for you.