Overlapping Named Entities & pattern approach

[julianpollmann]

Hey,
I've got a use case where I can have overlapping entities in my custom component, which I handle with filter_spans. This has the disadvantage that some of the entities are not matched. I saw that this can be solved by SpanCategorizer.
Since my approach is for now pattern based, how can I implement my custom component to use SpanCategorizer? Do I need to write a custom Suggester?

Custom Pipeline Component:

def multi_token_ner(doc):
  patterns = [
      {"label": "CompleteMatch", "pattern": "X1 G4 D3"},
      {"label": "PartialMatch", "pattern": "D3"}
  ]

  original_ents = list(doc.ents)
  found_ents = []

  for pattern in patterns:
      for match in re.finditer(pattern['pattern'], doc.text):
          start, end = match.span()
          span = doc.char_span(start, end)
          if span is not None:
              found_ents.append((span.start, span.end, span.text))

      for ent in found_ents:
          start, end, name = ent
          ent_span = Span(doc, start, end, label=pattern['label'])
          original_ents.append(ent_span)

      filtered = filter_spans(original_ents)
      doc.ents = filtered

  return doc

Sample Input data:

...
X1 G4 D3

Expected Entities:

1st Entity (ent.text, ent.label_): X1 G4 D3 CompleteMatch
2nd Entity (ent.text, ent.label_): D3 PartialMatch

Thanks in advance!

[thomashacker]

Hello ✨,
you can save the spans inside doc.spans instead of doc.ents , giving you the possibility to have overlapping spans. (https://spacy.io/api/doc#spans)

For your usecase it could look like this:

doc.spans["Match"] = original_ents

doc.spans is a dictionary of SpanGroups which is a container for Span objects, instead of the key "Match", used in the example above, you can choose whatever key you seem fit. 😄 Since your use case works with a rule based approach, there is no need to implement the SpanCategorizer pipeline inside your function

[polm]

Also note we're working on a SpanRuler which should do exactly what you want, though it might not be released for a while yet.

#9880