custom component does not initialize

[riccardopinosio]

I have the following custom component in its own package. It is supposed to take a file of json patterns and then match these patterns with the DependencyMatcher

@Language.factory("dependency_matcher_extractor")
def make_dependency_matcher_extractor(nlp: Language, name: str):
    return DependencyMatcherExtractor(nlp, name)

class DependencyMatcherExtractor(Pipe):
    '''Implements relation extraction with dependency matching
    '''
    def __init__(self, nlp: Language, name: str = "dependency_matcher_extractor", patterns = {}) -> None:
        # create the matcher
        self.nlp = nlp
        self.name = name
        self.matcher = DependencyMatcher(nlp.vocab)
        self.patterns = patterns

        # populate the relation extractors patterns for each relation name based on pattern file
        for relation_name in self.patterns:
            relation_patterns = self.patterns[relation_name]
            self.matcher.add(relation_name, relation_patterns)

        # register custom extension
        if not Doc.has_extension("dependency_matches"):
            Doc.set_extension("dependency_matches", default=[])

    def __call__(self, doc: Doc) -> Doc:
        # add the matches when doc is processed
        pdb.set_trace()
        print(self.patterns)
        matches = self.matcher(doc)
        doc._.dependency_matches.append(matches)
        return doc

    def to_disk(self, path, exclude=tuple()):
        path = ensure_path(path)
        if not path.exists():
            path.mkdir()
        srsly.write_json(path / "patterns.json", self.patterns)
    
    def from_disk(self, path, exclude=tuple()):
        self.patterns = srsly.read_json(path / "patterns.json")
        return self

    def initialize(self, get_examples=None, nlp=None, patterns: Dict = {}):
        self.patterns = patterns

This component is exposed to the conda environment as an entry point:

setup(
    name='customs',
    version='0.1.0',
    install_requires=[
        'wheel',
        'pydantic',
        'srsly',
        'spacy==3.2' # spacy version
    ],
    entry_points={
        "spacy_factories": ["dependency_matcher_extractor = src.dependency_matcher_extractor:make_dependency_matcher_extractor"]
    }
)

Then in my spacy.cfg I have:

[nlp]
lang = "en"
pipeline = ["tok2vec", "tagger", "parser", "attribute_ruler", "lemmatizer", "dependency_matcher_extractor", "entity_ruler", "ner", "merge_entities"] 
batch_size = 256
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.dependency_matcher_extractor]
factory = "dependency_matcher_extractor"

[...]

[initialize.components]

[initialize.components.entity_ruler]

[initialize.components.entity_ruler.patterns]
@readers = "srsly.read_jsonl.v1"
path = "/path_to_patterns"

[initialize.components.dependency_matcher_extractor]

[initialize.components.dependency_matcher_extractor.patterns]
@readers = "srsly.read_json.v1"
path = "/path_to_patterns"

Unfortunately, when I run the pipeline with the train command, it appears that the initialize method of the component is not run/triggered. Therefore, the patterns attribute is always the empty dictionary. I am not sure what's going on as the code I have seems identical to the examples on how to do this e.g. at https://spacy.io/usage/processing-pipelines#custom-components, and also similar to how the entity ruler is implemented (https://github.com/explosion/spaCy/blob/master/spacy/pipeline/entityruler.py), which in the above pipeline loads the patterns just fine. I'm not sure what's going on so any help is appreciated.

[adrianeboyd]

I think the problem is that your initialize method is only storing the patterns as an attribute and not processing them further. For a cleaner design with v3 (some of the entity ruler design is a holdover from v2), I'd recommend removing patterns as a kwarg from __init__ and moving the pattern loading to a separate add_patterns-type method that you can call with initialize. Or if you never need to support adding more patterns later, you can put the pattern loading directly in initialize if you want.

[riccardopinosio]

@adrianeboyd that makes a lot of sense, indeed yesterday I was able to make it work by copying the structure of the entityRuler. Now I have in the initialize method:

def initialize(self, get_examples=None, nlp=None, patterns = {}) -> None:
      self.patterns = patterns
      self._populate_patterns()

where the _populate_patterns() method does exactly what you describe. With this approach it works.

However, if this is the case, the documentation at https://spacy.io/usage/processing-pipelines#component-data seems a bit misleading. Wouldn't it be better to modify it so that the need for the _populate_pattern() method is made explicit? At least I tried to base my custom components off of those instructions and it wasn't clear to me that this was needed to make it work.

[adrianeboyd]

Yeah, what I said about loading in initialize isn't good advice for this case. When you're deserializing in from_disk you need to reload the patterns, too, so a separate method is better. Or alternatively you can design self.patterns as a property that does the loading in the setter. (In turn, this wouldn't be great for incremental pattern loading like what's supported by entity ruler.)

Thanks for the suggestion about the docs, we'll have a look! (PRs are also always welcome!)

[riccardopinosio]

@adrianeboyd I can write a PR for the docs to make this more explicit. Will take a look later this week. I assume from reading the contributing guidelines that I would first create an issue and then fork, change the docs, create PR?

[adrianeboyd]

You don't need to create an issue first, just fork+PR is fine!