Yet another "Can't find model" question

[phdavis1027]

I am trying to load the model fr_dep_news_trf inside a of a scrapy pipeline step. I have run python -m spacy download fr_dep_news_trf with the relevant virtualenv activated and verified that a dir called site-packages/fr_dep_news_trf exists in that virtualenv.

python -m spacy info gives this:

============================== Info about spaCy ==============================

spaCy version    3.7.6
Location         /home/phillipdavis/.pyenv/versions/sabaq/lib/python3.11/site-packages/spacy
Platform         Linux-6.2.6-76060206-generic-x86_64-with-glibc2.35
Python version   3.11.1
Pipelines        en_core_web_trf (3.7.3), fr_dep_news_trf (3.7.2)

python -m spacy validate gives this:

================= Installed pipeline packages (spaCy v3.7.6) =================
ℹ spaCy installation:
/home/phillipdavis/.pyenv/versions/sabaq/lib/python3.11/site-packages/spacy

NAME              SPACY            VERSION
en_core_web_trf   >=3.7.2,<3.8.0   3.7.3   ✔
fr_dep_news_trf   >=3.7.0,<3.8.0   3.7.2   ✔

python -m pip list gives:

fr-dep-news-trf            3.7.2
...
spacy                      3.7.6
spacy-curated-transformers 0.2.2
spacy-legacy               3.0.12
spacy-loggers              1.0.5

Most questions of this genre seem to boil down to IDE interference or some virtualenv problem. I'm only using the terminal, but I suppose it could still be a virtualenv problem. That said, I'm really at a loss for what I've messed up here. Would really appreciate any help.

[phdavis1027]

I solved this problem by writing this pair of small utility functions:

from spacy.language import Language as SpacyLanguage
import spacy.util as spacy_util

def find_path_to_site_packages() -> str | None:
    for p in sys.path:
        if "site-packages" in p:
            return p
    return None

def load_spacy_model(model: str) -> SpacyLanguage:
    version = spacy_util.get_package_version(model)
    cfg_dir = f"{model}-{version}"

    spacy_model_path = os.path.join(
            find_path_to_site_packages(), 
            model,
            cfg_dir)

    return spacy.load(spacy_model_path)