de_dep_news_trf: Installing the german pretrained transformer model seems to break my spacy installation

[drumkid96]

Hi!

I am encoutering some weird behaviour when installing the model "de_dep_news_trf".
It stated that this model had been trained on spacy version 3.7.0, which i tried as well.
In both cases it didnt seem to work.
I tried to validate the package, which lead to error messages.
I then removed my virtual environment, reinstalled via poetry from my .toml file with spacy[transformers, lookups] version 3.7.2 (the same as the german transformer model) but no luck.

I added the terminal output with the errors below.
Probably there is something i havent understood yet, as im fairly new to using spacy.
Hope someone can help me!

Cheers,
Rasmus

Edit 1:
I was using Python 3.12.5 and spaCy 3.7.2 (and therefore de_dep_news_trf 3.7.2)
Downgrading to Python 3.11.8, spaCy 3.6.1 (and therefore de_dep_news_trf 3.6.1) resulted in the package validating succesfully (de_dep_news_trf >=3.6.0,<3.7.0 3.6.1 ✔)

Terminal Log:

(metaphorai v0-2-py3.12) PS C:\Users\reber\iCloudDrive\1 Rasmus\10 VSCode\MetaphorAI V0.2> python -m spacy download de_dep_news_trf
Collecting de-dep-news-trf==3.7.2
Downloading https://github.com/explosion/spacy-models/releases/download/de_dep_news_trf-3.7.2/de_dep_news_trf-3.7.2-py3-none-any.whl (410.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.5/410.5 MB 1.6 MB/s eta 0:00:00
Requirement already satisfied: spacy<3.8.0,>=3.7.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from de-dep-news-trf==3.7.2) (3.7.2)
Collecting spacy-curated-transformers<0.3.0,>=0.2.0 (from de-dep-news-trf==3.7.2)
Using cached spacy_curated_transformers-0.2.2-py2.py3-none-any.whl.metadata (2.7 kB)
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (3.0.12)
Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (1.0.5)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (1.0.10)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (2.0.8)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (3.0.9)
Requirement already satisfied: thinc<8.3.0,>=8.1.8 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (8.2.5)
Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (1.1.3)
Requirement already satisfied: srsly<3.0.0,>=2.4.3 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (2.4.8)
Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (2.0.10)
Requirement already satisfied: weasel<0.4.0,>=0.1.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (0.3.4)
Requirement already satisfied: typer<0.10.0,>=0.3.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (0.9.4)
Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (6.4.0)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (4.66.5)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (2.32.3)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (2.9.1)
Requirement already satisfied: jinja2 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (3.0.3)
Requirement already satisfied: setuptools in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (74.1.2)
Requirement already satisfied: packaging>=20.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (24.1)
Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (3.4.0)
Requirement already satisfied: numpy>=1.19.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (1.26.4)
Collecting curated-transformers<0.2.0,>=0.1.0 (from spacy-curated-transformers<0.3.0,>=0.2.0->de-dep-news-trf==3.7.2)
Using cached curated_transformers-0.1.1-py2.py3-none-any.whl.metadata (965 bytes)
Collecting curated-tokenizers<0.1.0,>=0.0.9 (from spacy-curated-transformers<0.3.0,>=0.2.0->de-dep-news-trf==3.7.2)
Using cached curated_tokenizers-0.0.9-cp312-cp312-win_amd64.whl.metadata (2.0 kB)
Requirement already satisfied: torch>=1.12.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from spacy-curated-transformers<0.3.0,>=0.2.0->de-dep-news-trf==3.7.2) (2.4.1)
Requirement already satisfied: regex>=2022 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from curated-tokenizers<0.1.0,>=0.0.9->spacy-curated-transformers<0.3.0,>=0.2.0->de-dep-news-trf==3.7.2) (2024.7.24)
Requirement already satisfied: language-data>=1.2 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (1.2.0)
Requirement already satisfied: annotated-types>=0.6.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (0.7.0)
Requirement already satisfied: pydantic-core==2.23.3 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (2.23.3)
Requirement already satisfied: typing-extensions>=4.6.1 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (4.12.2)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (3.8)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (1.26.20)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (2024.8.30)
Requirement already satisfied: blis<0.8.0,>=0.7.8 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (0.7.11)
Requirement already satisfied: confection<1.0.0,>=0.0.1 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (0.1.5)
Requirement already satisfied: filelock in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from torch>=1.12.0->spacy-curated-transformers<0.3.0,>=0.2.0->de-dep-news-trf==3.7.2) (3.16.0)
Requirement already satisfied: sympy in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from torch>=1.12.0->spacy-curated-transformers<0.3.0,>=0.2.0->de-dep-news-trf==3.7.2) (1.13.2)
Requirement already satisfied: networkx in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from torch>=1.12.0->spacy-curated-transformers<0.3.0,>=0.2.0->de-dep-news-trf==3.7.2) (3.3)
Requirement already satisfied: fsspec in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from torch>=1.12.0->spacy-curated-transformers<0.3.0,>=0.2.0->de-dep-news-trf==3.7.2) (2024.9.0)
Requirement already satisfied: colorama in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from tqdm<5.0.0,>=4.38.0->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (0.4.6)
Requirement already satisfied: click<9.0.0,>=7.1.1 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from typer<0.10.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (8.1.7)
Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from weasel<0.4.0,>=0.1.0->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (0.16.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from jinja2->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (2.1.5)
Requirement already satisfied: marisa-trie>=0.7.7 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.0->de-dep-news-trf==3.7.2) (1.2.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\users\reber\appdata\local\pypoetry\cache\virtualenvs\metaphorai_v0-2-lwx5r2j--py3.12\lib\site-packages (from sympy->torch>=1.12.0->spacy-curated-transformers<0.3.0,>=0.2.0->de-dep-news-trf==3.7.2) (1.3.0)
Using cached spacy_curated_transformers-0.2.2-py2.py3-none-any.whl (236 kB)
Using cached curated_tokenizers-0.0.9-cp312-cp312-win_amd64.whl (731 kB)
Using cached curated_transformers-0.1.1-py2.py3-none-any.whl (25 kB)
Installing collected packages: curated-tokenizers, curated-transformers, spacy-curated-transformers, de-dep-news-trf
Successfully installed curated-tokenizers-0.0.9 curated-transformers-0.1.1 de-dep-news-trf-3.7.2 spacy-curated-transformers-0.2.2

[notice] A new release of pip is available: 24.1 -> 24.2
[notice] To update, run: python.exe -m pip install --upgrade pip
✔ Download and installation successful
You can now load the package via spacy.load('de_dep_news_trf')

(metaphorai v0-2-py3.12) PS C:\Users\reber\iCloudDrive\1 Rasmus\10 VSCode\MetaphorAI V0.2> python -m spacy validate
Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\spacy_main.py", line 4, in
setup_cli()
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\spacy\cli_util.py", line 84, in setup_cli
registry.cli.get_all()
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\catalogue_init.py", line 110, in get_all
result.update(self.get_entry_points())
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\catalogue_init_.py", line 125, in get_entry_points
result[entry_point.name] = entry_point.load()
^^^^^^^^^^^^^^^^^^
File "C:\Users\reber\AppData\Local\Programs\Python\Python312\Lib\importlib\metadata_init_.py", line 205, in load
module = import_module(match.group('module'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\reber\AppData\Local\Programs\Python\Python312\Lib\importlib_init_.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in find_and_load_unlocked
File "", line 935, in load_unlocked
File "", line 995, in exec_module
File "", line 488, in call_with_frames_removed
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\spacy_curated_transformers\cli\debug_pieces.py", line 20, in
from ..pipeline.transformer import CuratedTransformer
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\spacy_curated_transformers\pipeline_init.py", line 1, in
from .transformer import CuratedTransformer
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\spacy_curated_transformers\pipeline\transformer.py", line 24, in
from ..models.listeners import ListenerStateUtils
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\spacy_curated_transformers\models_init.py", line 1, in
from .architectures import (
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\spacy_curated_transformers\models\architectures.py", line 30, in
from ..tokenization.types import Tok2PiecesModelT
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\spacy_curated_transformers\tokenization_init.py", line 3, in
from .hf_loader import build_hf_piece_encoder_loader_v1
File "C:\Users\reber\AppData\Local\pypoetry\Cache\virtualenvs\metaphorai_v0-2-lWx5r2j--py3.12\Lib\site-packages\spacy_curated_transformers\tokenization\hf_loader.py", line 12, in
transformers.BertTokenizerFast,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'transformers' has no attribute 'BertTokenizerFast'

[honnibal]

Thanks, there's a stale requirement in the model. Looking into it.