Large model can't get initialize ( memoryview is too large)

[shmouelsamares]

While trying to load a custom model, I'm guetting the ValueError: memoryview is too large
The model is indeed very large ( vocab takes around 9GB) and the error seems to come from "msgpack_packer"

Is there a way to overcome this issue? or even better, to compress my language model?

[adrianeboyd]

So msgpack does have a hard-coded 2GB filesize limit for an individual file. I think the main place this could come up in practice is with a large transformer model. What components do you have in your pipeline (in particular, I'd be surprised if any of the files in the vocab/ folder are that large, maybe it's a different component)?

[shmouelsamares]

Hi,
My model contain only the vocab and NER componants.
The vectors file (txt UTF-8) indeed mesure 9GB. This model come from the Tech-Net project and I can't open it overview it as it is too large

Any idea as how to overcome this limit?

[adrianeboyd]

Hmm, you're loading the model from disk with spacy.load? If you're saving the model with nlp.to_disk the only part of the vectors that's stored as msgpack is vocab/key2row, so I'm not exactly sure what might be going on at this point. Or are you using nlp.to_bytes at some point?

Can you post the output of spacy info --markdown?

Can you post the traceback for the error when you run spacy.load?

Did you convert the vectors with spacy init vectors? And then trained with spacy train?

Can you show the file sizes for the files in model/vocab?

But there are good ways to reduce the size of the vectors. With 9GB of vectors you might be waiting a while or run out of RAM (I don't think I've tested this with more than ~2GB), but spacy does have built-in methods to prune vectors that work quite well in practice. We use them to create the vectors for the md models, which are the lg vectors pruned to 20K entries. You can use spacy init vectors to prune to a specified size: https://spacy.io/api/cli#init-vectors. Similar vectors are combined so that you keep the same vocab but don't have as many entries in the vector table.

[shmouelsamares]

i first converted the vectors with spacy init vectors, then i'm loading the model from disk with spacy.load

the markdown return:

## Info about spaCy

- **spaCy version:** 3.5.1
- **Platform:** Windows-10-10.0.22621-SP0
- **Python version:** 3.10.5
- **Pipelines:** en_coreference_web_trf (3.4.0a0), en_core_web_lg (3.4.0), en_core_web_md (3.5.0), en_core_web_sm (3.5.0), en_core_web_trf (3.5.0), fr_core_news_sm (3.5.0)

the file size of model/vocab/vectors is 9GB, I wouln't know how to prune it wisely

[adrianeboyd]

Can you post the traceback for the error? (I still don't immediately see how this would result in a msgpack error from the code on our side, but I might be overlooking something.)

I'd strongly suggest trying spacy init vectors --prune, it sounds drastic but it really works quite well without reducing the vocab. I know that it can prune 500k vectors to 20k vectors in <10 minutes and the performance hit is usually very small even for this level of pruning. If you keep a larger number of vectors (maybe 1M as a starting point for your testing?) then the performance hit may be even smaller.

[shmouelsamares]

Could you please explain what the prune parameter is doing? is it limiting the dimension of the vectors?

[adrianeboyd]

Pruning collapses similar vectors, so two similar vectors like "cat" and "kitty" might point to the same row in the vectors data after pruning. It doesn't modify the dimension.

[shmouelsamares]

I have executed the --prune init command and am waiting for the task to be completed,

here's the error for the initialize command:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-20-4c47fa030deb> in <module>
      1 #memoryview(nlp)
----> 2 nlp.initialize()

~\AppData\Roaming\Python\Python36\site-packages\spacy\language.py in initialize(self, get_examples, sgd)
   1285         try:
   1286             init_vocab(
-> 1287                 self, data=I["vocab_data"], lookups=I["lookups"], vectors=I["vectors"]
   1288             )
   1289         except IOError:

~\AppData\Roaming\Python\Python36\site-packages\spacy\training\initialize.py in init_vocab(nlp, data, lookups, vectors)
    134     # warn if source model vectors are not identical
    135     sourced_vectors_hashes = nlp.meta.pop("_sourced_vectors_hashes", {})
--> 136     vectors_hash = hash(nlp.vocab.vectors.to_bytes(exclude=["strings"]))
    137     for sourced_component, sourced_vectors_hash in sourced_vectors_hashes.items():
    138         if vectors_hash != sourced_vectors_hash:

~\AppData\Roaming\Python\Python36\site-packages\spacy\vectors.pyx in spacy.vectors.Vectors.to_bytes()

~\AppData\Roaming\Python\Python36\site-packages\spacy\util.py in to_bytes(getters, exclude)
   1309 
   1310 def to_bytes(getters: Dict[str, Callable[[], bytes]], exclude: Iterable[str]) -> bytes:
-> 1311     return srsly.msgpack_dumps(to_dict(getters, exclude))
   1312 
   1313 

~\AppData\Roaming\Python\Python36\site-packages\spacy\util.py in to_dict(getters, exclude)
   1327         # Split to support file names like meta.json
   1328         if key.split(".")[0] not in exclude:
-> 1329             serialized[key] = getter()
   1330     return serialized
   1331 

~\AppData\Roaming\Python\Python36\site-packages\spacy\vectors.pyx in spacy.vectors.Vectors.to_bytes.serialize_weights()

~\AppData\Roaming\Python\Python36\site-packages\srsly\_msgpack_api.py in msgpack_dumps(data)
     12     RETURNS (bytes): The serialized bytes.
     13     """
---> 14     return msgpack.dumps(data, use_bin_type=True)
     15 
     16 

~\AppData\Roaming\Python\Python36\site-packages\srsly\msgpack\__init__.py in packb(o, **kwargs)
     53     Pack an object and return the packed bytes.
     54     """
---> 55     return Packer(**kwargs).pack(o)
     56 
     57 

~\AppData\Roaming\Python\Python36\site-packages\srsly\msgpack\_packer.pyx in srsly.msgpack._packer.Packer.pack()

~\AppData\Roaming\Python\Python36\site-packages\srsly\msgpack\_packer.pyx in srsly.msgpack._packer.Packer.pack()

~\AppData\Roaming\Python\Python36\site-packages\srsly\msgpack\_packer.pyx in srsly.msgpack._packer.Packer.pack()

~\AppData\Roaming\Python\Python36\site-packages\srsly\msgpack\_packer.pyx in srsly.msgpack._packer.Packer._pack()

~\AppData\Roaming\Python\Python36\site-packages\srsly\msgpack\_packer.pyx in srsly.msgpack._packer.Packer._pack()

ValueError: memoryview is too large

[adrianeboyd]

Do you have a custom merging tokenizer or do your input texts contain tokens like information_processing_method? If not, then you have a whole row of vector data that will never be used by the spacy model, and if you have individual words like information processing method, the vectors for those individual words won't have been trained on this context for the embeddings.

[shmouelsamares]

Yes I do.
However I understand you point and I agree this is a painfully unoptimized method.

I noticed that Spacy NER has some multi-token tagging capabilities, would it be possible to "summarize" all the multi-words tokens of my database by refering to their components?

[adrianeboyd]

So just on the technical side, if you really want to try this out, you can try commenting out this section in spacy/training/initialize.py:

spaCy/spacy/training/initialize.py

Lines 136 to 139 in dbc71ec

	vectors_hash = hash(nlp.vocab.vectors.to_bytes(exclude=["strings"]))
	for sourced_component, sourced_vectors_hash in sourced_vectors_hashes.items():
	if vectors_hash != sourced_vectors_hash:
	warnings.warn(Warnings.W113.format(name=sourced_component))

If you're not sourcing components in your config this check doesn't matter (and we'll look at restricting it more in a future release, since it's also slow if you don't need the checks).

[adrianeboyd]

This has been updated in #12607, to be released in v3.5.3.