SpanRuler removes all existing labels if overwrite is set to True

[NixBiks]

If I add a SpanRuler with annotate_ents and overwrite set to True then all existing entities are removed from the document. I'd expect that it would keep the existing but overwrite the ones that would overlap with the new patterns.

I'd expect the following code to pass

import spacy

text = "Apple is opening its first big office in San Francisco."
nlp = spacy.blank("en")
ruler = nlp.add_pipe(
    "span_ruler", config={"validate": True, "spans_key": None, "annotate_ents": True}
)
patterns = [
    {"label": "ORG", "pattern": "Apple"},
    {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]},
]
ruler.add_patterns(patterns)
assert len(nlp(text).ents) == 2

nlp.add_pipe(
    "span_ruler",
    name="span_ruler2",
    config={
        "validate": True,
        "spans_key": None,
        "annotate_ents": True,
        "overwrite": True,
    },
)
assert len(nlp(text).ents) == 2  # <- this fails because it now has length 0 instead

I'm not sure if this is intended or not?

[NixBiks]

Ahh it's probably intended. Instead I should not overwrite but replace the ents_filter with {"@misc": "spacy.prioritize_new_ents_filter.v1"}

The following works

import spacy

text = "Apple is opening its first big office in San Francisco."
nlp = spacy.blank("en")
ruler = nlp.add_pipe(
    "span_ruler", config={"validate": True, "spans_key": None, "annotate_ents": True}
)
patterns = [
    {"label": "ORG", "pattern": "Apple"},
    {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}]},
]
ruler.add_patterns(patterns)
assert len(nlp(text).ents) == 2

ruler2 = nlp.add_pipe(
    "span_ruler",
    name="span_ruler2",
    config={
        "validate": True,
        "spans_key": None,
        "annotate_ents": True,
        "overwrite": False,
        "ents_filter": {"@misc": "spacy.prioritize_new_ents_filter.v1"},
    },
)
ruler2.add_patterns([
    {"label": "ABC", "pattern": "Apple"},
])
doc = nlp(text)
assert len(doc.ents) == 2
assert doc.ents[0].label_ == "ABC", doc.ents[0].label_

[adrianeboyd]

If you're using spans_key=None, you can just use the entity_ruler instead if you'd prefer. In spacy v3 the entity_ruler is still a completely separate component, but in spacy v4 the entity_ruler will be implemented as a SpanRuler in the background and use these ents filters to keep the same behavior as before. So in v4 you will also be able to use entity_ruler with the same options/behavior as before.

[NixBiks]

Yeah. I like the flexibility with SpanRuler though. In fact I extend the SpanRuler to allow for additional features, ie. having the ability to parse the match different expressed when creating the patterns. A little complex

[adrianeboyd]

Yeah, it's definitely way more flexible in terms of how the spans are sorted and filtered.