Proposal: Non-NER span categorizer

[honnibal]

People often want to train spaCy's EntityRecognizer component on sequence labeling tasks that aren't "traditional" Named Entity Recognition (NER). A named entity actually has a fairly specific linguistic definition: they're either definite descriptors, or a numeric expression. Software designed for doing NER may not perform well on other sequence labelling tasks. This is especially true of spaCy's NER system, which makes fairly aggressive trade-offs. What we need is another system that's better suited for these non-NER sequence tasks.

WIP can be found here: https://github.com/explosion/spaCy/blob/feature/span-categorizer/spacy/pipeline/spancat.py

Examples of non-NER span categorization

(This section needs to be filled out with actual examples. For now I've just noted some general categories of problem.)

• Chatbot tutorials and software have taken to calling all intent arguments "entities". I wish they wouldn't, but they do --- so people want to recognize and label these phrases with an "entity recognizer". These phrases could be anything. They're whatever argument of the "intent" the author has defined, even if they don't really make good linguistic sense. The boundaries of these "entities" can also be quite inconsistently defined.

• In aspect-oriented sentiment analysis, one often has a set of attributes to recognize, like "price", "quality", "customer service", etc. Sometimes these can be recognized as topics in a sentence, but often people want them localised to specific spans.

• In information or relation extraction, it's often important to recognize predicate words or phrases. For verbs, this is often a single word, but sometimes you want nouns as well, which can be multi-word expressions.

• Tasks such as disfluency detection, detection of spelling errors, detection of grammatical errors etc are all quite sensible to code as sequence tagging problems.

Problems with IOB/BILUO

NER systems typically make one decision per word. Usually that decision corresponds to a tag (such as an IOB or BILUO tag). spaCy does things only slightly differently: on each word we predict an action, where the actions correspond to the BILUO tags. Encoding the task as transition-based parsing has some subtle advantages I won't go into here.

The problem is, the BILUO-style approach implies a loss where a boundary error means the model's prediction is completely wrong. This is the right way to think about NER and probably chunking, but it doesn't match up well with what people often want on non-NER problems. Often people are pretty ambivalent about the exact boundaries of their target spans. They would much rather have the boundary off-by-one than miss the entity entirely.

Suggested solution

The SpanCategorizer will consist of three components:

• User-configurable functions to suggest candidate spans from a Doc
• A submodel to extract token vector features (e.g. the current CNN, a BiLSTM, etc)
• A submodel to classify the spans

For instance, we might set out get_spans function to get the span offsets for all unigrams, bigrams, noun chunks and named entities. No matter how many candidate spans we have, we only need to run the token-to-vector model once -- and that model should be able to do most of the "heavy lifting". For each span, we need to perform some sort of weighted sum, and then run a feed-forward network. Hopefully this can be quite cheap.

There's an additional trick we could play when encoding the problem, to reduce the number of spans. If we have relatively few classes (especially if we only have one class), we could have extra classes that denote span offsets. This is what the object detection systems like YOLO do. Instead of generating all the overlapping spans, we can generate non-overlapping spans, and have classes that tell us to adjust the border. This might make things easier for the model as well, if it means the spans are of more uniform length.

If we do have spans of non-uniform length, we should still be able to process them efficiently. We need to be able to handle cases where there are some long spans, and a lot of short ones, without blowing up the space by padding out into a huge square. Batching the spans by length internally should take care of this.

I think it would be good to have a loss function that assigns partial credit to spans according to how many tokens of overlap they have with the gold-standard. For instance, if we extract a span with 90% overlap with a gold-standard phrase, we can make the loss small.

Current progress

• Sketch out pipeline component
• Sketch out architecture for span-to-scores model
• Write utility functions for component the tok2vec and span-to-scores models
• Test utility functions
• Implement dummy experiment 1 as test: replace some words with TARGET, try to detect them.
• Implement dummy experiment 2 as test: Try to recognize two-word sequences that start with the dummy word TARGET, or three word sequences with TARGET in the middle.
• Implement script for NER experiment
• Implement script for chunking experiment
• Find dataset for more motivated experiment, implement script.
• Tweak the model to get better results on experiments.
• Write example
• Document model

[niedakh]

Don't know if this will help, but I've come to understand that one of the key factors is that the non-NER span detection task is vastly different in terms of detecting whether to start a span vs whether to end it.

Usually a decision to start tagging a sequence is relatively easy, there's a sequence of words that bring us to a place in the text which becomes a placeholder to express something, think of looking for a topic introduction phrase in a limited domain context.

On the other hand the decision whether to close the span is extremely hard. Has enough information been said? Is the meaning of the text shifting? Especially in compound sentences or even multi-sentence spans.

Example: if you are doing a zoning process in an urban municipal office, you will get a lot of input from people living in the area. You can imagine that there is a need to detect spans of text containing information about requirements and needs that people have with an area. You will usually get an easy tasks of finding where they start: There should be a place to relax. denoting the need for a, duh, place to relax, is fairly easy. But generalizing where a span should end in cases like: I must put my bike somewhere, as this is usually a third step on my shopping route. I will have a lot of things in my sack. is not that easy. The actual need is a bike parking spot for a person who performs shopping and carries some of the items on the bike, so the annotator would select a span put ... sack as THE_NEED label.

[honnibal]

the annotator would select a span put ... sack as THE_NEED label.

Well wait a second here! Yeah an annotator might initially want to do that, but you definitely can't let them. That's hopelessly broken.

Even aside from the recognition challenge, if the span scope and content is so undefined, how do you even use it in the rest of your application? You'll have this chunk of text and you'll have to like, make another bag of words model or something. You're setting yourself an unsolvable NLP problem that will take ages to annotate (as your spans are really long and uncertain), only to queue up a second difficult NLP problem downstream.

[niedakh]

Yes, the task does require an extra model in many cases, but actually is a reasonable expectation in a non-NER information extraction scenario from your list of use cases in the proposal. Don't chatbots require extra models to handle non-NER arguments to their intents?

I agree that this is a complicated task, ideally the annotator would have a wider ontology that allows splitting THE_NEED into multiple specific cases. But I do think that this is a fair general use case / expectation of a non-NER span marking model, to use it as tier-one approach for area of interest extraction in text. This area of interest might be quite complicated to define and many users might want to define it naively, by just tagging the maximum sequence that conveys the desired meaning. My example is an extreme case of the multi-word expression, but I don't think the use case is. I also came across this much more in conversations than in written text. What do you think? Is this in-scope?

[honnibal]

I wouldn't normally be so direct, but since we got to know each other a bit at IRL: No no, that's all wrong. You definitely shouldn't want this.

If you need to extract a general area of interest, you should work at the sentence or paragraph level and just do text classification. Trying to extract a long, vaguely-defined region of text as a span is a terrible bad no good idea. You may as well align it to a sentence or paragraph boundary, because you're not getting any value out of the subsentence information, and it makes the task basically impossible, both to annotate and to recognise.

This is also totally different from what's called a "multi-word expression". A MWE is basically a "word with spaces" --- a term like "artificial intelligence" is best understood as a single concept, rather than as a sum of its parts.

[ramji-c]

I think it might also be useful to have a variant of the loss, that works on a vector similarity rather than % string overlap. For instance, in free text queries, multi-token names can be considered as some perturbed version of a canonical name. In such cases, a loss that works with the vector representation of the spans and the gold-standard might be better than a loss that works with surface string overlap.

[honnibal]

@ramji-c that's a very interesting idea. I'll think about that.

[EtienneAb3d]

You may have look at my chunker / entity extractor here (using spaCy):
https://github.com/EtienneAb3d/OpenNeuroSpell
It can be tested here:
http://lextract.cubaix.com/

More tools here (spell corrector, semantic grouping, ..):
http://cubaix.com/

Coming very soon: a Cross-lingual Semantic Memory = feed sentences in any language, search sentences having a similar meaning to a given sentence in any other language.

[kommerzienrat]

I am working on a comparable issue and I am stuck with a solution for extracting the phrases. AFAIK, there are mostly solutions that expect a noun to be a central aspect of the phrase. In my current domain (service reviews), this does not help. Noun chunking is not doing the job and neither are dependency or constituency parsing. What approach do you have in mind? (I mean, do you extract all phrases, even overlapping ones?)

[EtienneAb3d]

@kommerzienrat, can you provide with some of your sentences to test ?

[kommerzienrat]

Indeed, e.g. the following sentence is a good example:
"The physician was very friendly and explained everything very well even though his explanations could have had more details, as he did not take a lot of time for us, but the receptionist was unfriendly." --> Classes in this sentence are: friendliness, explanation(i.e. information by the doctor), time usage for patient; But: They are all represented by different constructions and kinds of words, such as verbs, adjectives, nouns... I am currently annotating data with prodigy. But annotations in the field of spans are, as mentioned above, quite tricky. Not to mention how to find those spans in documents in order to classify them.

[EtienneAb3d]

Are you looking for something like this ?

If yes, as said above, have a look at my chunker here (using spaCy):
https://github.com/EtienneAb3d/OpenNeuroSpell

[kommerzienrat]

It does have to do with my problem but seems not to be a perfect match for such spans.
For example, what is more, I am dealing with German data mostly and there exist a lot of cases such as verbal constructs that are torn apart in the sentence. It is like 'He took his [more text] time.' Besides, I want to extract spans like 'very friendly', 'explained everything' and I am still not sure whether 'explained everything' or 'explained everything very well' should be extracted.

[Henry-E]

@kommerzienrat you could check out Flair framework which does this kind of sequence labelling and is easy to get working.https://github.com/zalandoresearch/flair/blob/master/README.md

On a separate note, do you have any recommendations for guidelines about constructing the kind of ontology needed to decide on useful spans?

[tokestermw]

yay thanks for this 💯.

So for my use case, the span indices are pre-determined using some rule. So I would think a loss function that's just cross entropy would be more apt. It would be nice to have as an option as part of the config. If I have time, I'll test an example.

The default for _get_all_spans seems to enumerate all possible span indices. It seems like you should use an NER model if you have to do this. If the span indices also have to be predicted, I've seen AllenNLP use a parametrized span pruner to map span indices -> word embeddings -> prune probabilities.

[kommerzienrat]

@kommerzienrat you could check out Flair framework which does this kind of sequence labelling and is easy to get working.https://github.com/zalandoresearch/flair/blob/master/README.md

On a separate note, do you have any recommendations for guidelines about constructing the kind of ontology needed to decide on useful spans?

Thanks for the link to flair. It is good to see more great stuff coming from Europe/Germany. I regret that it might not help in my case, but anyway, I am still having a try. What is more, is that I cannot determine span boundaries due to the nature of my data. I would love to see a solution as proposed in the first post above where one missing word does not prevent a whole span from being used.

[niedakh]

I think that a decent use case (and a data set) would be these kinds of de-anonymization efforts where the model aims at detecting a span containing NER-like personal information: https://ai.google/tools/datasets/audio-ner-annotations/

Some are single-word and clearly NER like a person's name, others are relevant numbers that have personal meaning like a health insurance identifier or the name of the hospital a person goes to.

[kommerzienrat]

@honnibal Is there yet something we could try to use? I am working on a comparable issue (aspect-based SA) and still have not found a working solution to identify spans in texts. Thanks in advance!

[svlandeg]

More recent work-in-progress: #6747

Update: will be released as an experimental functionality in spaCy 3.1!