Make spacy models and extra package download compatible with PEP 503

[evan0greenup]

Currently, the models package should be installed via python -m spacy download <model_package>, It cannot be directly installed via pip install.

Why not provide a single plain html page under the domain spacy.io which provide the links compatible with PEP503.

As a reference, pytorch-cuda is a example of this see pytorch cuda116.

The desired models installation method should be like pip install --extra-index-url <xxx>.spacy.io/<vvv>/models/ or other similar format.

It would be easy to make the package manager like pip, poetry or other high level package manager based on pip more operable.

[evan0greenup]

As spaCy is an industrial ready NLP framework, package management should be easy and automatic and also compatible with popular pip and other common package manager.

[adrianeboyd]

To clarify, the model packages are valid pip packages and they are compatible with pip, poetry, etc.:

pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.0/en_core_web_sm-3.4.0-py3-none-any.whl

As part of setup.cfg:

install_requires =
    en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.0/en_core_web_sm-3.4.0-py3-none-any.whl

Or more securely including the checksum:

install_requires =
    en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.0/en_core_web_sm-3.4.0-py3-none-any.whl#sha256=35365d49d389ecb19ba33119dca9ab9ff80b36fbeb063fa5d44008197fead8fa

And the docs include a related example with the #egg format and a .tar.gz archive: https://spacy.io/usage/models#production

The checksums are included at the top of the release notes, e.g.: https://github.com/explosion/spacy-models/releases/tag/en_core_web_sm-3.4.0

You can follow the "Release details" button on any model entry in https://spacy.io/models/ to get to the release notes, which include the .whl and .tar.gz packages as release assets, or you can search under https://github.com/explosion/spacy-models/releases/.

---

I think there are two separate issues:

• The direct URLs / dependency specifications are too hard to figure out. We are aware of this and we're discussing some options in Add a way to get the URL to download a pipeline to the CLI #11175.
• I would be worried that an index server for the models could lead to a lot of unwanted behavior.

In terms of the second issue, this install command would be fine, where you'd end up with the most recent version of the model and a compatible version of spacy:

pip install en_core_web_sm --extra-index-url https://...

Or a specific model version, which would currently end up with spacy==3.3.1:

pip install en_core_web_sm==3.3.0 --extra-index-url https://...

However, someone who runs:

pip install spacy==3.2.4
pip install en_core_web_sm --extra-index-url https://...

... is not going to end up with the version of spacy that they intended. They'll currently get en_core_web_sm==3.4.0 and it will clobber their whole spacy install as part of the process.

Someone who runs:

pip install spacy==3.0.7 en_core_web_lg --extra-index-url https://...

... is going to end up downloading the models for v3.4.0, v3.3.0, v3.2.0, and v3.1.0 before pip finds a version compatible with the spacy version. This is over 2GB of unnecessary downloads, and just gets worse over time.

I'm not sure that there's a good solution given the limitations of pip, but suggestions are welcome.