packaging data with a custom language component

[DSLituiev]

I am trying to add a custom language component to an existing NER pipeline.
First, I added it in a python script, and saved to_disk.

My custom component file, postprocess.py looks like this:

import json
import logging

import spacy
from spacy.language import Language
from spacy.tokens import Span

Span.set_extension("cui", default=None)
Span.set_extension("snomed_ct", default=None)
Span.set_extension("canonical_text", default=None)



with open("mapping-ontology-to-umls.json") as fh:
    meta = json.load(fh)


@Language.component("sdoh_cui")
def add_sdoh_cui(doc):
     # ... set extension attributes based on mappings above

Now I have a config.cfg:

[paths]
train = "./data/spacy-level-2/sdoh-cocoa-skippy-no-empty-dev.spacy"
dev = "./data/spacy-level-2/sdoh-cocoa-skippy-no-empty-test.spacy"
vectors = null
init_tok2vec = null
code = "./postprocess.py"

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","ner","sdoh_cui"]
batch_size = 100
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"

[components.ner.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 96
attrs = ["NORM","PREFIX","SUFFIX","SHAPE","SPACY"]
rows = [5000,2500,2500,2500,100]
include_static_vectors = false

[components.ner.model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.sdoh_cui]
factory = "sdoh_cui"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 96
attrs = ["NORM","PREFIX","SUFFIX","SHAPE","SPACY"]
rows = [5000,2500,2500,2500,100]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 2000
max_epochs = 30
max_steps = 0
eval_frequency = 200
frozen_components = ["tok2vec"]
before_to_disk = null
annotating_components = []

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.0002

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = null
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
after_init = null
before_init = {"@callbacks":"customize_tokenizer_pathology"}

[initialize.components]

[initialize.lookups]
@misc = "spacy.LookupsDataLoader.v1"
lang = ${nlp.lang}
tables = ["lexeme_norm"]

[initialize.sdoh_cui]

[initialize.tokenizer]

In setup.py I've added these lines:

from setuptools import setup

model_name = "en_sdoh"
setup(
    name=model_name,
    entry_points={
        "spacy_factories": ["sdoh = postprocess:add_sdoh_cui"]
    },
    package_data={model_name:
        ["functions.py", "postprocess.py", "mapping-ontology-to-umls.json",],
)

When I package it with python -m spacy package ./sdoh-best ./packages --code sdoh-best/postprocess.py,sdoh-best/functions.py,sdoh-best/mapping-ontology-to-umls.json,
I am running into:

ℹ Building package artifacts: sdist
Traceback (most recent call last):
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/click/core.py", l
ine 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/spacy/cli/package.py", line 47, in package_cli
    package(
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/spacy/cli/package.py", line 94, in package
    util.import_file(code_path.stem, code_path)
  File "/Users/dlituiev/anaconda3/envs/nlp3.9/lib/python3.9/site-packages/spacy/util.py", line 1313, in import_file
    spec.loader.exec_module(module)  # type: ignore[union-attr]
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "sdoh-best/postprocess.py", line 14, in <module>
    with open("mapping-ontology-to-umls.json") as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'mapping-ontology-to-umls.json'

I tried adding this to the config.cfg, but that didn't help:

[components.sdoh_cui.data]
@readers = "srsly.read_json.v1"
path = "mapping-ontology-to-umls.json"

[adrianeboyd]

If you're using a separate package with entry points, you would install this package with pip and you wouldn't need to include these files with --code. In this case, I think you need to figure out how to refer to a path based on the install directory, which you can do with __file__.

If it's not an entry point in a separate package, then the typical method would be to load the data from a kwarg setting in Pipe.initialize (path specified with a reader in something like [initialize.components.component_name.setting]) and then always serialize this data with the component in to/from_disk/bytes so the path isn't needed after the one-time initialization step, which is run as part of spacy train or spacy assemble.

If you try to use both --code and the installed package with the same factory name, you should see errors about the name being registered more than once, so check what's going on there. (If something is misconfigured, entry points can fail silently, so you may need to do some debugging.)

[DSLituiev]

Thank you for your help!
I have code and data. The code is not an entry point. The code seemingly gets included, but data JSON file does not.

I was trying to follow documentation here, which suggested that solution with [components.component_name.data]. I tried adding that data reader here

[initialize.components.component_name.setting]
@readers = "srsly.read_json.v1"
path = "mapping-ontology-to-umls.json"

but that didn't help.

I am using component -- maybe that is the problem. I see more info about the factory, but not the component serialization.

[DSLituiev]

If you try to use both --code and the installed package with the same factory name,
This is definitely not the case.

I tried moving the data file before the proper code files in --code comma separated list, and in that case it breaks, because --code tries to evaluate or import the code, but what I am trying to include is a JSON file.

When I remove it, I'm seeing again the issue with failure to find the data file, even though it's listed under both initialize and components.

[initialize.components.component_name.setting]
@readers = "srsly.read_json.v1"
path = "mapping-ontology-to-umls.json"


[components.sdoh_cui.data]
# 🚨 Problem: Runtime pipeline depends on local path
@readers = "srsly.read_json.v1"
path = "mapping-ontology-to-umls.json"

Do you have an example of using data with a component (as opposed to factory)?

[DSLituiev]

Even a factory would do the job. I see many documentation pieces here, but I believe I am not getting something fundamentally. A reproducible end-to-end code would be much appreciated.

[DSLituiev]

So here is what helped. This video suggest putting all factories into an __init__.py in a certain hierarchy location. Here is an alternative solution

The solution consists of these steps:

1. python -m spacy package trained_model target
2. Go to the target directory. There will be a following maze of confusingly similarly named directories:
   en_modelname-0.0.0/modelname/en_modelname-0.0.0

i. The first level has setup.py -- leave as is.
ii. The second level en_modelname-0.0.0/modelname has__init__.py
iii. The third level en_modelname-0.0.0/modelname/en_modelname-0.0.0 has files that were included with --code, including a file with factories (let's call it factories.py) and directories corresponding to each pipeline component.

3. The file with the factories in the third level is not imported. To make it visible, we move it to the second level, and import within the __init__.py: from .factories import *
4. The data files may be packaged in the third level. It needs to be moved even further within the pipeline component so that config can point to it.
5. Make sure the component is included in en_modelname-0.0.0/modelname/en_modelname-0.0.0/config.cfg, e.g.

[nlp]
lang = "en"
pipeline = ["tok2vec","ner", "mycomponent"]

...
[components.mycomponent]
factory = "myfactory"

[initialize.mycomponent]

We did not list data in the config file, but it still worked well.
6. Place factory data into en_modelname-0.0.0/modelname/en_modelname-0.0.0/myfactory folder
7. Go to the first level en_modelname-0.0.0/ and serialize the model python setup.py sdist. This will produce a file in dist/*.tar.gz
8. Install the model with pip install dist/en_modelname-0.0.0.tar.gz

[adrianeboyd]

I can see that that would work in the end, but you're doing a lot of steps by hand that you can implement as part of the component pretty easily.

I'm not 100% sure what happens with a component registered as a factory, but I suspect you could get the component solution to work in the external package with entry points if you loaded the data with paths relative to __file__ somehow. It's not going to work with --code, which just imports python code and doesn't handle other types of files.

But in general, managing additional packages is a hassle, so I would pick the following option if I were implementing this, with the data loaded from a path in initialize and the data saved out by the serialization methods, since JSON-serializable data is very easy to save and load:

https://spacy.io/usage/processing-pipelines#component-data-initialization

Then you only need to include the code with --code.

Another example with the standard initialization and serialization is the component in this PR, which loads patterns from JSONL: https://github.com/explosion/spaCy/pull/9880/files

And I suppose the more hacky option if the JSON file isn't that large is to convert it to a native python data structure and import it directly.