Unable to load GPU trained model on machine without a GPU

[pifov]

Hello everyone,

Currently I am trying to deploy a spacy model which is trained on a GPU due to contained "NER" and "Transformer" in its pipeline. Then the model is pickled using dill and I am trying to load it on a kubernetes pod where there is no GPU available.

Using spacy 3.4.1 and python 3.7

Interestingly before we were able to surpass this obsticle by setting spacy to "spacy.require_cpu" before pickling it. However now we can't due to an error during serialiasing:

ValueError                                Traceback (most recent call last)
<ipython-input-21-a36d6cd6798c> in <module>
     10 # LogsCollector.log("INFO","Saving the model...",)
     11 with open(model_local_path, 'wb') as f:
---> 12     dill.dump(main_model, f)
~/anaconda/envs/mario_env/lib/python3.7/site-packages/dill/_dill.py in dump(obj, file, protocol, byref, fmode, recurse, **kwds)
    334     _kwds = kwds.copy()
    335     _kwds.update(dict(byref=byref, fmode=fmode, recurse=recurse))
--> 336     Pickler(file, protocol, **_kwds).dump(obj)
    337     return
    338 
~/anaconda/envs/mario_env/lib/python3.7/site-packages/dill/_dill.py in dump(self, obj)
    618             raise PicklingError(msg)
    619         else:
--> 620             StockPickler.dump(self, obj)
    621         return
    622     dump.__doc__ = StockPickler.dump.__doc__
~/anaconda/envs/mario_env/lib/python3.7/pickle.py in dump(self, obj)
    435         if self.proto >= 4:
    436             self.framer.start_framing()
--> 437         self.save(obj)
...
---> 47             raise ValueError("Cannot convert non-numpy from base Ops class")
     48 
     49     def minibatch(
ValueError: Cannot convert non-numpy from base Ops class

The following issue appears when I am trying to load the model on a CPU machine:

  File "C:\work\git-repos\reader-models-api\app\load_models.py", line 54, in init_models
    spacy_models[model_type] = dill.load(open(path, 'rb'))
  File "C:\work\_virtual_environments\entity-reader39\lib\site-packages\dill\_dill.py", line 373, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "C:\work\_virtual_environments\entity-reader39\lib\site-packages\dill\_dill.py", line 646, in load
    obj = StockUnpickler.load(self)
  File "C:\work\_virtual_environments\entity-reader39\lib\site-packages\dill\_dill.py", line 636, in find_class
    return StockUnpickler.find_class(self, module, name)
ModuleNotFoundError: No module named 'cupy'

Appreciate your help!

[polm]

The issue here is that because you're using dill it tries to recreate the same objects, not equivalent ones. If the model is created on GPU then you have cupy tensors, but on a system without a GPU you don't have cupy, so there's no way to create the same object.

What you should do in this case is use nlp.to_disk("some-path") to save the pipeline, and spacy.load("some-path") to load it.

If it's easier to use dill for some reason, then you can save the model after GPU training and load it on the same machine after calling require_cpu - that should get the types right for serialization.

For more details or other ways to serialize it, see the serialization docs.

[pifov]

Hi I have decided to take the second aproach and train first with GPU and load it after spacy.require_cpu() is called. However I'm experiancing different kind of issue. When training the model with a custom config I can't save it to disk. The custom config looks like this:

        config_ner = {
            "components": {
                "transformer":{
                    "model":{
                    "@architectures": "spacy-transformers.TransformerModel.v3",
                    "name": "custom_transformer", # need to put correct path when fit_tranformer ok
                    "mixed_precision": False,
                    "get_spans": {
                    "@span_getters": "spacy-transformers.strided_spans.v1",
                    "window": 128,
                    "stride": 96
                }}
                }
            }
        }
        self.model_level_1.add_pipe("ner", last=True)
        self.model_level_1.add_pipe(
            "transformer", config=config_ner)
        ner = self.model_level_1.create_pipe("ner", "transformer")

I'm not hundred percent sure where the problem is.
Again I appreciate your help.

[polm]

When you say you can't save the model to disk, what do you mean? Do you get an error?

The contents of your config look OK, but there are some issues with the other lines - the transformer should be before the NER component, so you should add it first. Since components are added at the end by default, you'll end up with [ner, transformer] with your current code. Also, you need to configure the NER component to listen to the transformer - by default it'll use an internal tok2vec instead. You can create a GPU config with the training widget for an example of a config with a listener.

I'm also not sure why you're calling create_pipe, that shouldn't be necessary, and based on this code sample, isn't doing anything.