NER with only the keyword?

[haydenso]

It appears that spaCy's NER only works accurately when the keyword is blended in a sentence, are there any methods to get accurate NER results by training a model with only the keyword? Or is it best to formulate random words to put the keyword in a sentence with?

[polm]

The golden rule of training data is that the more it is like your real data, the better your results will be.

When reading a sentence, the context around the entities can be as important as the entities themselves. For example, if you say, "I went to XXX for vacation", we can guess that XXX is a location. If your training data doesn't have any of that context it will be hard to learn from. Building sentences with "random words" will not help.

If you only have a list of keywords you may be able to use a rule based matcher or do weak supervision.

[haydenso]

Hey! Thank you so much for your help!

For extra information, I have a series of research paper abstracts, along with a list of medical terms. I have taken about half (600+) of the abstracts and using a rule-based matcher, found the medical term's position. I then formatted it and exported it as a Doc object, and trained it in a NER model, hoping it'll be able to identify other medical terms.

I now have a model that is able to identify the position of the keywords, however, it seems as though it only matches up ones that appear in the list I have already. How do I improve it such that it is able to detect ones that are beyond the finite list I used? Would using a revision dataset (e.g ones with ORG, PERSON, DATE etc.) and then combining it into one model be the best approach?

[polm]

For medical abstracts and medical keywords I don't think combining it with standard entities will help much, because I don't think there will be a lot of PERSON/ORG/LOCATION, and when they're there I don't think it'll be ambiguous.

600 abstracts is not a huge corpus, so you probably need more data. I guess a question is:

1. How big is your list of terms?
2. How diverse is your list of terms? Do the words look similar?
3. How diverse is the real world data you're trying to match?
4. What is your reported accuracy on a dev set in training?

[haydenso]

I have more abstracts available to test, but would there be an issue if the corpus is also the list of text I am then going to run the model with also?

The list of terms is about 639 long, but the list of terms are all gene symbols that basically have the same format - perhaps this is the problem.

The scores are shown:
================================== Results ==================================

TOK 100.00
NER P 100.00
NER R 100.00
NER F 100.00
SPEED 14845

[polm]

Are you evaluating on your training data? You should have separate train and dev data.

It seems like your model is definitely overfitting, so I would strongly recommend you annotate a little data by hand so that your data includes more diverse tokens.