Factory from_disk not called during initialization

[DSLituiev]

I am trying to assemble a pipeline from several components that work separately.

python -m spacy assemble --code all_code.py config.cfg package/

and getting an error that shows that one of the components has not been initialized with from_disk

AttributeError: 'Level2Predictor' object has no attribute 'matcher'

Here are relevant parts of the config

[nlp]
lang = "en"
pipeline = ["tok2vec","parser","senter","attribute_ruler","lemmatizer","ner","sdoh_bow", "sentencizer", "sdoh_cui"]
disabled = ["senter"]
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 256
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components.sdoh_bow]
factory = "sdoh_bow"

[components.sdoh_cui]
factory="sdoh_cui"

[initialize.components.sdoh_bow]
path = "en_sdoh_bow/sdoh_bow"

[initialize.components.sdoh_cui]
path = "data/"

and the factory:

@Language.factory("sdoh_bow")
class Level2Predictor():

    def __init__(self, nlp, name="sdoh_level2"):
        self.nlp = nlp
        return

    def from_disk(self, path, exclude=tuple()):
        path_matcher = path / "domain_patterns_dict_manedit.json"
        path_tfidf = path / "tfidf.pickle"
        self.matcher = initalize_matcher(path_matcher, self.nlp)
        with open(path_tfidf, "rb") as fh:
            self.tfidf = pickle.load(fh)
        self.classification_models = read_classification_models(path)

    def to_disk(self, path, exclude=tuple()):
        path_matcher = path / "domain_patterns_dict_manedit.json"
        path_tfidf = path / "tfidf.pickle"

        matcher_patterns = {self.matcher.vocab.strings[key]:val
                            for key, val in self.matcher._patterns.items()}
        with open(path_matcher, "w") as fh:
            json.dump(matcher_patterns, fh)

        with open(path_tfidf, "wb") as fh:
            pickle.dump(self.tfidf, fh)
        write_classification_models(path, self.classification_models)


    def __call__(self, doc):
        matches = self.matcher(doc)
        spans = [Span(doc, start, end, label=match_hash) for match_hash, start, end in matches]
        spans = filter_spans(spans)
        spans = self.predict_spans(spans)
        doc.set_ents(spans)
        return doc

and I do have the data file in the specified path: en_sdoh_bow/sdoh_bow/domain_patterns_dict_manedit.json

What am I missing here?

[adrianeboyd]

spacy assemble (and spacy train) initialize components with initialize, not with from_disk. The idea is that data can be loaded from a local system path once in initialize (and this is very flexible with registered functions as readers), and from that point on the serialization methods save and load from bytes/directories without requiring any external paths. See: https://spacy.io/usage/training#initialization. Some examples are the patterns in EntityRuler or the lookups in Lemmatizer.

[DSLituiev]

is there an example of how to write and use initialize with custom factories and how to pass custom data to them?

In my case, I would not need any more complexity beyond from_disk. I still would need to figure out how to format the path and pass it to the initialize().

[DSLituiev]

I've found this example:

https://spacy.io/usage/processing-pipelines#component-data-initialization

I followed the example by feeding my JSON data file and a JSON reader in the cfg file.

[nlp]
lang = "en"
pipeline = ["tok2vec","ner","sdoh_cui"]
batch_size = 100
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.sdoh_cui]
factory = "sdoh_cui"
# ...

[initialize]
# ...

[initialize.components.sdoh_cui]

[initialize.components.sdoh_cui.data]
# ✅ This only runs on initialization
@readers = "srsly.read_json.v1"
path = "sdoh-with-cuis/sdoh_cui/mapping-ontology-to-umls.json"

The postprocess.py

@Language.factory("sdoh_cui")
class SDOH:
    # ...
    def initalize(self, get_examples=None, nlp=None, data={}):
        logging.info("initializing sdoh_cui")
        logging.info(f"data: {data}")
        self.data = data

Then I run an assembly script:

python -m spacy assemble     config-cui.cfg     $CUI_MODEL_DIR     --code sdoh-with-cuis/postprocess.py     --components.ner.source en_sdoh_cnn_ner

Still, my component remains without data when I run the module

From this message, I see that the sdoh_cui is not being initialized despite initialize.components.sdoh_cui.data section in the config file :

=========================== Initializing pipeline ===========================
tokenizer customized
[2022-05-13 12:51:16,647] [INFO] Added vocab lookups: lexeme_norm
[2022-05-13 12:51:16,647] [INFO] Created vocabulary
[2022-05-13 12:51:17,920] [INFO] Added vectors: en_core_web_md
[2022-05-13 12:51:18,025] [INFO] Finished initializing nlp object
✔ Initialized pipeline

[adrianeboyd]

I think this is a case where it's hard to know without a complete example. I tested the example from the docs and it worked fine with spacy assemble:

from spacy.language import Language
from spacy.util import ensure_path


@Language.factory("acronym")
class AcronymComponent:
    def __init__(self, name):
        self.data = {}

    def __call__(self, doc):
        return doc

    def initialize(self, get_examples=None, nlp=None, data={}):
        self.data = data

    def to_disk(self, path, exclude=tuple()):
        path = ensure_path(path)
        if not path.exists():
            path.mkdir()
        srsly.write_json(path / "data.json", self.data)

    def from_disk(self, path, exclude=tuple()):
        self.data = srsly.read_json(path / "data.json")
        return self

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["acronym"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.acronym]
factory = "acronym"

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
tag_acc = 1.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.acronym]

[initialize.components.acronym.data]
@readers = "srsly.read_json.v1"
path = "/tmp/data.json"

[initialize.tokenizer]