How to add models to a blank nlp pipeline?

[lingvisa]

Spacy: 3.3.0
Lang: Chinese

I defined a custom language and add my own syntax_iterator for phrase extraction as below:

from .syntax_iterators import SYNTAX_ITERATORS
from .tag_map import TAG_MAP
class CustomChineseDefaults(Chinese.Defaults):
    stop_words = STOP_WORDS
    syntax_iterators = SYNTAX_ITERATORS
    tag_map = TAG_MAP

@spacy.registry.languages("custom_zh")
class CustomChinese(Chinese):
    lang = "custom_chinese"
    Defaults = CustomChineseDefaults

Then I am trying to use the CustomChinese this way:

import spacy
from wbnlu.chinese.custom_chinese import CustomChinese

source_nlp = spacy.load("zh_core_web_lg")
#print(source_nlp._path, source_nlp.pipe_names)
nlp = CustomChinese()
nlp.tokenizer = source_nlp.tokenizer
pipeline = ["tok2vec", "tagger", "ner", "attribute_ruler"]
for name in pipeline:
    nlp.add_pipe(name, source=source_nlp)

This seems to work, but it prints out this warning message, which is concerning:

/Users/congminmin/nlp/kbm-ner/venv/lib/python3.7/site-packages/spacy/language.py:710: UserWarning: [W113] Sourced component 'tok2vec' may not work as expected: source vectors are not identical to current pipeline vectors.
  warnings.warn(Warnings.W113.format(name=source_name))
/Users/congminmin/nlp/kbm-ner/venv/lib/python3.7/site-packages/spacy/language.py:710: UserWarning: [W113] Sourced component 'tagger' may not work as expected: source vectors are not identical to current pipeline vectors.
  warnings.warn(Warnings.W113.format(name=source_name))
/Users/congminmin/nlp/kbm-ner/venv/lib/python3.7/site-packages/spacy/language.py:710: UserWarning: [W113] Sourced component 'ner' may not work as expected: source vectors are not identical to current pipeline vectors.
  warnings.warn(Warnings.W113.format(name=source_name))
/Users/congminmin/nlp/kbm-ner/venv/lib/python3.7/site-packages/spacy/language.py:710: UserWarning: [W113] Sourced component 'attribute_ruler' may not work as expected: source vectors are not identical to current pipeline vectors.
  warnings.warn(Warnings.W113.format(name=source_name))

Also, I tried to load the models from disk, but it only loads tokenizer:
nlp.from_disk('/Users/congminmin/nlp/kbm-ner/venv/lib/python3.7/site-packages/zh_core_web_lg/zh_core_web_lg-3.3.0')

So how to fix my issue in loading models from my blank nlp pipeline object?

[adrianeboyd]

The warning is a serious warning. It means that the sourced components won't work because the vectors are missing.

You need to also copy the vectors from zh_core_web_lg along with the components. In a config (also with assemble) you'd specify this in [initialize] as vectors = "zh_core_web_lg".

If you're doing this in a script then you should be able to just copy the vectors (with v3.2+):

nlp.vocab.vectors = source_nlp.vocab.vectors

[lingvisa]

Yes, that warning message went away after copying the vectors.

[lingvisa]

I got another error:

File "/Users/congminmin/nlp/kbm-ner/wbnlu/ne_ml/ne_test.py", line 28, in chunks
    for np in doc.noun_chunks:
  File "spacy/tokens/doc.pyx", line 866, in noun_chunks
NotImplementedError: [E894] The 'noun_chunks' syntax iterator is not implemented for language 'zh'.

It seems my CustomChinese inheritance didn't work. I used to directly add a syntax_iterator.py into Spacy's source code for the Chinese language and it worked. Now I didn't modify spacy source and this way should be more elegant to add things to the language I want. Can you see my custom_chinese.py definition? It looks right.

[lingvisa]

Yes, this fixed it. and the problem seems to be the copy of tokenizer, which instead should be initialized from a config file. One more question regarding the initialization time of the components in two ways:

1. source_nlp = spacy.load("zh_core_web_lg")

pipeline = ["tok2vec", "tagger", "ner", "attribute_ruler"]
    for name in pipeline:
      nlp.add_pipe(name, source=source_nlp)

The first way takes 1 second to load all the components, while the 2nd takes 12 seconds to load the 4 components specified. Due to the Language customization, the loading time may not be improved?

Also, if I try:
nlp.tokenizer.initialize(pkuseg_model="mixed")
It gives this warning message:
WARNING: features.msgpack does not exist, try loading features.pkl

[adrianeboyd]

If you're concerned about the speed, you can save the new pipeline to disk and then the loading time should be similar to zh_core_web_lg.

The warning is because mixed is an original pkuseg model (from their repo, not ours) that uses pickle rather than our spacy-pkuseg model, which switch from pickle to msgpack. If you save and reload the model it should load without warnings.

[lingvisa]

“If you save and reload the model it should load without warnings”： spacy_ontonotes is the spacy version of "mixed"? The original pkusege has several models for different domains of text.

[adrianeboyd]

The original pkuseg models are trained on a wider variety of texts. For licensing reasons, we only distribute a model that we trained just on OntoNotes (which we have a license for). mixed may be slightly better, but we don't want to redistribute it with the zh_core_web_* models.

When you load a pkuseg model with spacy-pkuseg, spacy-pkuseg loads the data from the original pickle format. When the model is saved with nlp.to_disk spacy-pkuseg always saves the data as msgpack. So you should only see the warning the first time it's loaded.

[lingvisa]

What's the right way to load 'mixed' model using spacy-pkuseg? This only loads 'ontonotes': nlp.tokenizer.initialize(pkuseg_model="spacy_ontonotes")

[lingvisa]

If I save my CustomChinese pipleline into a location:

source_nlp = spacy.load("zh_core_web_lg")
cfg = {"segmenter": "pkuseg"}
nlp = CustomChinese.from_config({"nlp": {"tokenizer": cfg}})
nlp.tokenizer.initialize(pkuseg_model="spacy_ontonotes")
nlp.vocab.vectors = source_nlp.vocab.vectors
pipeline = ["tok2vec", "tagger", "ner", "attribute_ruler"]
for name in pipeline:
    nlp.add_pipe(name, source=source_nlp)

Now save it:

nlp.to_disk('../resources/models/spacy')

Then try to load it back:

cfg = {"segmenter": "pkuseg"}
nlp = CustomChinese.from_config({"nlp": {"tokenizer": cfg}})
nlp.tokenizer.initialize(pkuseg_model="spacy_ontonotes")
 nlp.from_disk('../resources/models/spacy')

This doesn't load any models, however. The pipeline is empty except for the tokenizer itself. The nlp.components is an empty list, so no components will be loaded from the model path.

[adrianeboyd]

You should reload saved models with spacy.load().

If you use spacy.load() with a directory path, you have to manage importing the additional CustomChinese code yourself in your script.

If you package the model with spacy package --code code.py and install this package, then loading the model from the package name (not a path) will load the provided code automatically.

[lingvisa]

I executed this command at the project root:

(venv) congminmin@congmins-MacBook-Pro-2 kbm-ner % ls ner/chinese                                                                                           
__init__.py             __pycache__             custom_chinese.py       stop_words.py           syntax_iterators.py     tag_map.py

Then I do:
(venv) congminmin@congmins-MacBook-Pro-2 kbm-ner % spacy package --code ner/chinese ner/resources/models/spacy ner/resources/models

Then I got this error message:

ℹ Building package artifacts: sdist
Traceback (most recent call last):
  File "/Users/congminmin/nlp/kbm-ner/venv/bin/spacy", line 8, in <module>
    sys.exit(setup_cli())
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/spacy/cli/package.py", line 49, in package_cli
    package(
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/spacy/cli/package.py", line 96, in package
    util.import_file(code_path.stem, code_path)
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/spacy/util.py", line 1352, in import_file
    module = importlib.util.module_from_spec(spec)  # type: ignore[arg-type]
  File "<frozen importlib._bootstrap>", line 553, in module_from_spec
AttributeError: 'NoneType' object has no attribute 'loader'

I tested the CustomChinese class, and it seems working well. So what am I missing in packaging the model?

[adrianeboyd]

--code expects a comma-separated list of paths to .py files. Here I think it would be easiest to combine everything into a single file.

[lingvisa]

What's the format of comma-separated list of paths? Since I have only one input directory and I don't need a comma, so I use "--code ner/chinese".

I also tried this but failed:

spacy package --code ner/chinese/__init__.py, ner/chinese/custom_chinese.py, ner/chinese/stop_words.py, ner/chinese/syntax_iterators.py, ner/chinese/tag_map.py, ner/chinese/tf_vector.py ner/resources/models/spacy ner/resources/models

[adrianeboyd]

It is comma-separated without spaces.

[lingvisa]

But this time it can't find the CustomChinese class when it is packaging:

Traceback (most recent call last):
  File "/Users/congminmin/nlp/kbm-ner/venv/lib/python3.8/site-packages/spacy/util.py", line 329, in get_lang_class
    module = importlib.import_module(f".lang.{lang}", "spacy")
  File "/usr/local/Cellar/[email protected]/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'spacy.lang.custom_chinese'

Re-copy the 'save' code of the CustomChinese model below:

start = time.time()
source_nlp = spacy.load("zh_core_web_lg")
print('Total time: ', int(time.time()-start))
print(source_nlp._path, source_nlp.pipe_names)
print('Loading custom nlp ...')
cfg = {"segmenter": "pkuseg"}
nlp = CustomChinese.from_config({"nlp": {"tokenizer": cfg}})
nlp.tokenizer.initialize(pkuseg_model="spacy_ontonotes")

print('Copy vectors and tokenizer ...')
nlp.vocab.vectors = source_nlp.vocab.vectors

nlp._path = source_nlp._path
pipeline = ["tok2vec", "tagger", "ner", "attribute_ruler"]
print('Add pipeline ...')
start2 = time.time()
for name in pipeline:
    nlp.add_pipe(name, source=source_nlp)
print('Load pipeline time: ', int(time.time()-start2))
nlp.to_disk('../resources/models/spacy')

The generated meta.json contains:

{
  "lang":"custom_chinese",
  "name":"pipeline",
  "version":"0.0.0",
  "spacy_version":">=3.4.0,<3.5.0",
  "description":"",
  "author":"",
  "email":"",
  "url":"",
  "license":"",
  "spacy_git_version":"d583626a8",
  "vectors":{
    "width":300,
    "vectors":500000,
    "keys":500000,
    "name":"zh_vectors",
    "mode":"default"
  },
  "labels":{
    "tok2vec":[

    ],

[adrianeboyd]

For each file included, e.g. with the name my_code.py, this needs to work when the file is in your current working directory:

from . import my_code

As I said above, I'd recommend creating a single file with all your registered functions with a name like code.py (in particularly not with a name like __init__.py, which may be causing some of the errors you're seeing).

To make sure the basics are working you can test it like this:

from . import code
import spacy
nlp = spacy.blank("custom_chinese")

Here's a minimal example: https://spacy.io/usage/linguistic-features#language-subclass