Has anyone PyInstaller working for Spacy with CUDA?

[mbrunecky]

As it is, using PyInstaller with Spacy is not that difficult. That is, unless you need to have import spacy_legacy, which is not handled by contributed hook-spacy.py. It gets harder when you want to use transformers. Again, a custom hook-transformers.py can be used to import another 8 modules or so.
But using transformers without GPU is a bit slow. So I need a CUDA support, and may early attempts suggest that it will not be easy.

Has anyone used PyInstaller to build Spacy executable with CUDA support?
If so, can you share some highlights?

[svlandeg]

Hi! I don't think any of us have immediate experience with this. Maybe it would be worth asking the Pyinstaller community?

[mbrunecky]

I just managed to get it all work – with my Spacy 3.1 and NVIDIA CUDA 11.5. I will post the details (hook-spacy.py, hook-treansformers.py, hook-cupy.py) but right now have to deal with some medical. The trick is (among others) installing the _right_ combination of torch+cudannn, doing pip install is not enough, one MUST uninstall the ‘wrong’ versions repeatedly. Martin

[mbrunecky]

Here are the details of how I managed to get it to work. Note that it may not work for you, because your venv may already contain some conflicting modules/versions. Neither is it a good 'pythonic' example.
Note the final distribution using transformers is a monster - 7.7 GB (including en_core_web_trn and _lg). I gave up on using PyInstaller option --onefile because building the archive takes forever (and so does the first invocation). Besides, my distribution needs some other files, so using --onedir and then zip-ing it is far more convenient and faster.

My environment:

spaCy version    3.2.0
Location         C:\Work\ML\Spacy3\lib\site-packages\spacy
Platform         Windows-10-10.0.19042-SP0
Python version   3.9.7
Pipelines        en_core_web_lg (3.2.0), en_core_web_md (3.2.0), en_core_web_sm (3.2.0), en_core_web_trf (3.2.0)

Most important is getting the 'right' combination of torch and cupy. My CUDA is 11.5 (NVIDIA did not give me much choice - the older releases seem to be well hidden). In my venv, I had to REPATEDLY uninstall any 'not so current versions' of torch and cupy. Note I started with cupy-cuda114 which was not 'right':

pip uninstall cupy-cuda114
pip install -U cupy-cuda113
pip uninstall torch
pip install torch==1.10.0+cu113   -f https://download.pytorch.org/whl/torch_stable.html

Then, when using PyInstaller (the most recent version), I had to add the following hooks. Note PyInstaller comes with a huge lists of hooks, including one for Spacy - but when you add your own, it uses them both. Hooks go into hooks directory specified on command line (default is ./)l

1. hook-spacy.py Because I use some 3.1 trained models, I had to import spacy_legacy:

# HOOK FILE FOR SPACY
from PyInstaller.utils.hooks import collect_data_files, collect_submodules, collect_all

# ----------------------------- SPACY -----------------------------
print('=================== SPACY =====================')
datas = collect_data_files("spacy")
hiddenimports = collect_submodules("spacy")

# ----------------------------- spacy_legacy -----------------------------
data = collect_all('spacy_legacy')
datas += data[0]
binaries = data[1]
hiddenimports += data[2]

# ----------------------------- spacy models -----------------------------
# data = collect_all('en_core_web_trf')
# datas += data[0]
# binaries = data[1]
# hiddenimports += data[2]

# data = collect_all('en_core_web_lg')
# datas += data[0]
# binaries = data[1]
# hiddenimports += data[2]

# alternatively, on PyInstaller command line you may just add:
# --collect-all=en_core_web_lg
# --collect-all=en_core_web_trf

print('=================== SPACY DONE =====================')

2. hook-transformers.py IF you are using transformers, there is a bunch of imports you need:

# HOOK FILE FOR SPACY TRANSFORMERS
from PyInstaller.utils.hooks import collect_all

# ----------------------------- TRANSFORMERS -----------------------
print('=================== TRANSFORMERS =====================')
data = collect_all('transformers')
datas = data[0]
binaries = data[1]
hiddenimports = data[2]
# There should be a way to get this list, perhaps from importlib_metadata
modules = ['tqdm', 'regex', 'sacremoses', 'requests', 'packaging', 'filelock', 'numpy', 'tokenizers' ]
for mod in modules:
    data = collect_all(mod)
    datas += data[0]
    binaries += data[1]
    hiddenimports += data[2]

# MXB: This is not the right way to write a hook file, but this gets transformers imports and it works.
# When transformers update, we will have to update the above list...

print('=================== TRANSFORMERS DONE =====================')

3. hook-cupy.py You need this if you want to use GPU (note transformers work without GPU, but you need a LOT of cores)

# HOOK FILE FOR CUPY
from PyInstaller.utils.hooks import collect_data_files, collect_submodules, collect_all

# ----------------------------- CUPY -----------------------------
print('=================== CUPY =====================')
data = collect_all('cupy')
datas = data[0]
binaries = data[1]
hiddenimports = data[2]

# ----------------------------- cupy_backends -----------------------------
data = collect_all('cupy_backends')
datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- fastrlock -----------------------------
data = collect_all('fastrlock')
datas += data[0]
binaries += data[1]
hiddenimports += data[2]

print('=================== CUPY DONE =====================')

[polm]

Thanks for sharing how you got this working!

[mbrunecky]

Unfortunately, there is a lot more 'fine detail'. The PyInstaller documentation helps, but it is not enough for a novice coping with a large project such as Spacy with transformers and torch.
Despite running from my venv, I have to explicitly add
--paths ../Lib/site-packages
Then, some command options take blank as the option/value separator, other command options require --option=value.
Interestingly, adding additional files to package described as
--add-data <SRC;DEST>
seems to require that DEST is a directory, so for adding my spacy train --code .\spacy_functions.py I had to use
--add-data=.\spacy_functions.py;spacy_functions
otherwise PyInstaller creates a directory 'spacy_functions.py' for my code.

Since including 'torch' makes the distribution huge (5+GB), I ended up generating two separate distributions: 'full' (with torch and transformers) and 'lite' (no torch, transformers ~2G only). And since the package can invoke one executable only - but I need to invoke multiple 'commands' from that package, I ended up writing a top level 'dispatch.py'. I invoke a specific script as
MyExe <what> <options...>
for example:
MyExe spacy <spacy cmd options>
Note that to run 'spacy train' using transformers one has to import also spacy_transformers (not needed to use trained models).

[qojjs]

I used your code in my project but it appeared another bug :
(ValueError: [E002] Can't find factory for 'transformer' for language Chinese (zh). This usually happens when spaCy calls nlp.create_pipe with a custom component name that's not registered on the current language class. If you're using a Transformer, make sure to install 'spacy-transformers'. If you're using a custom component, make sure you've added the decorator @Language.component (for function components) or @Language.factory (for class components).)
I tried to fix it with installing 'spacy-transformers' , but it didn't work and still have this problem . Could you please help me with this?
I used the model "zh_core_web_trf_3.2.0" and spaCy v 3.2.1

Another question is that do "zh_core_web_trf_3.2.0" really need 'spacy-transformers' ? I can use "zh_core_web_trf_3.2.0" in my project without pyinstaller's packing .

I hope i can get help , thanks a lot .

[mbrunecky]

This sounds like PyInstaller is not including spacy_transformers.
My code explicitly imports:
import spacy_transformers
import torch

Module spacy_transformers (I believe) registers the factory for 'transformer' pipeline component.
And, yes the naming convention "*trf_3.2.0" is used for transformer models.

[qojjs]

I think i catch your mind . Thanks a lot ! And i will go to try it again.

[Eschivo]

Maybe including CUDA dlls in the dist folder of the generated exe could also solve the issue. See these Github issues on the pyinstaller repo:
1, 2, 3