Prodigy NER Model to Spacy.load()

[tushitdave]

I have created annotated dataset using prodigy and ran prodigy model using below command:-

python -m prodigy train ner train_cv_model en_vectors_web_lg --n-iter 25 --output ./train_cv_model

and my model is being saved in my directory "Saved model: D:**\Resume Screening"

Now after this , I have decided to run model performance on my Jupyter notebook by using spacy , wherein I have given command
nlp = spacy.load('./train_cv_model')

Now I have a below error:-

C:\Users\TushitD\AppData\Roaming\Python\Python38\site-packages\spacy\util.py:707: UserWarning: [W095] Model 'en_vectors_web_lg' (2.3.0) requires spaCy >=2.3.0,<2.4.0 and is incompatible with the current version (3.0.0). This may lead to unexpected results or runtime errors. To resolve this, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
warnings.warn(warn_msg)

OSError Traceback (most recent call last)
in
----> 1 nlp = spacy.load('./train_cv_model')

~\AppData\Roaming\Python\Python38\site-packages\spacy_init_.py in load(name, disable, exclude, config)
45 RETURNS (Language): The loaded nlp object.
46 """
---> 47 return util.load_model(name, disable=disable, exclude=exclude, config=config)
48
49

~\AppData\Roaming\Python\Python38\site-packages\spacy\util.py in load_model(name, vocab, disable, exclude, config)
322 return load_model_from_package(name, **kwargs)
323 if Path(name).exists(): # path to model data directory
--> 324 return load_model_from_path(Path(name), **kwargs)
325 elif hasattr(name, "exists"): # Path or Path-like to model data
326 return load_model_from_path(name, **kwargs)

~\AppData\Roaming\Python\Python38\site-packages\spacy\util.py in load_model_from_path(model_path, meta, vocab, disable, exclude, config)
386 meta = get_model_meta(model_path)
387 config_path = model_path / "config.cfg"
--> 388 config = load_config(config_path, overrides=dict_to_dot(config))
389 nlp = load_model_from_config(config, vocab=vocab, disable=disable, exclude=exclude)
390 return nlp.from_disk(model_path, exclude=exclude)

~\AppData\Roaming\Python\Python38\site-packages\spacy\util.py in load_config(path, overrides, interpolate)
543 else:
544 if not config_path or not config_path.exists() or not config_path.is_file():
--> 545 raise IOError(Errors.E053.format(path=config_path, name="config.cfg"))
546 return config.from_disk(
547 config_path, overrides=overrides, interpolate=interpolate

OSError: [E053] Could not read config.cfg from train_cv_model\config.cfg

• which says english model [en_vectors_web_lg] is associated with spacy version of 2.3.4 or below however I am using spacy version 3.0.6 in my prodigy environment and spacy environment[This is where I am trying to load and run the model] both. If you try pip install spacy=3.0.6 version , that is to say english vector model automatically get downloaded as per the Spacy version.

• Also When I saved my model in my directory , I can't find any config.cfg file in the directory. Is this a external downloadable file. I cant find the same , which suppose to be associated and should be a part of prodigy NER model.

Please advise.

Versions:-
prodigy Version 1.10.7
Python Version 3.6.13
Database Name SQLite
Spacy version 3.0.6

[adrianeboyd]

Hi, prodigy v1.10 is definitely running with spacy v2.3.x and not spacy v3, so you'll need spacy v2.3.x (current release is v2.3.7) to load this model.

[tushitdave]

@adrianeboyd - My rest of the code part belong to version spacy version 3.0.6 and annotations are parsed from version 3.0.6 and part of same project. and this is where I have to load my NER model and run a test on rest of the dataset. Kindly suggest what needs to be done here

[adrianeboyd]

If you want to train a spacy v3 model with this data, then you need to use prodigy data-to-spacy to convert the data to the v2 JSON training format. Then with spacy v3, use spacy convert to convert the data to the v3 DocBin training format and spacy init config to create the right pipeline config to use with this data (probably spacy init config -p ner -o accuracy). Finally, train with spacy train.

This won't be 100% identical to the prodigy train model because the underlying model/parameters are slightly different and it's not the exact same vectors, but it should be very similar.

If it helps to see an example with the whole workflow, have a look at the pipelines/ner_demo project:

https://github.com/explosion/projects/tree/v3/pipelines/ner_demo

This demo has a custom converter instead of spacy convert, but otherwise the project steps are very similar.