Multiprocessing with doc containing custom attributes raises serialization error

[probavee]

Hello Spacy's community !!
I finished my spacy pipeline in which a custom attribute is attached to each doc.

class MyAttr:
  def __init__(self):
    self.elements:List[MySubAttr]

class MySubAttr:
  def __init__(self):
    self.matches: List[Tuple[str, int, int]]]
    self.matches2: List[Tuple[str, List[int]]
    self.range: int

class AssignAttr:
  def __init__(self, nlp):
    self.nlp = nlp
  def __call__(self, doc):
    doc._.my_attr = MyAttr()
    return doc

@spacy.language.Language.factory("assign_attr")
def create_multigram_component(
    nlp: spacy.language.Language, name: str
):  
    """create component multigram"""
    return AssignAttr(nlp)
spacy.tokens.Doc.set_extension("my_attr", default=MyAttr())

nlp = spacy.load("fr_core_news_lg")
nlp.add_pipe("assign_attr")

Everything is fine when i'm processing the pipeline but if I use nlp.pipe([text, ...], n_process>1) I got an TypeError MyAttr is not serializable.
I don't understand why this is happening. Do you have any clue ?

Thank you !

[ljvmiranda921]

Hi @ColleterVi , can you post the whole traceback for the TypeError? I tried your sample code and can't seem to reproduce the error.
Also, it seems that there's a typo in your MySubAttr class, you forgot to add a closing square bracket in self.matches2.

[probavee]

Hello !
Yes sorry for the typo, I also forgot to add the line code for set_extension, I fixed it.
Here is the whole traceback:

Traceback (most recent call last):
  File "/home/vee/Documents/octopulse/backend/deep_analysis/notebooks/main_dev.py", line 37, in <module>
    docs = list(doc)
  File "/home/vee/.local/lib/python3.10/site-packages/spacy/language.py", line 1575, in pipe
    for doc in docs:
  File "/home/vee/.local/lib/python3.10/site-packages/spacy/language.py", line 1639, in _multiprocessing_pipe
    self.default_error_handler(
  File "/home/vee/.local/lib/python3.10/site-packages/spacy/util.py", line 1617, in raise_error
    raise e
ValueError: [E871] Error encountered in nlp.pipe with multiprocessing:

Traceback (most recent call last):
  File "/home/vee/.local/lib/python3.10/site-packages/spacy/language.py", line 2188, in _apply_pipes
    byte_docs = [(doc.to_bytes(), doc._context, None) for doc in docs]
  File "/home/vee/.local/lib/python3.10/site-packages/spacy/language.py", line 2188, in <listcomp>
    byte_docs = [(doc.to_bytes(), doc._context, None) for doc in docs]
  File "spacy/tokens/doc.pyx", line 1287, in spacy.tokens.doc.Doc.to_bytes
  File "spacy/tokens/doc.pyx", line 1346, in spacy.tokens.doc.Doc.to_dict
  File "/home/vee/.local/lib/python3.10/site-packages/spacy/util.py", line 1259, in to_dict
    serialized[key] = getter()
  File "spacy/tokens/doc.pyx", line 1343, in spacy.tokens.doc.Doc.to_dict.lambda18
  File "/home/vee/.local/lib/python3.10/site-packages/srsly/_msgpack_api.py", line 14, in msgpack_dumps
    return msgpack.dumps(data, use_bin_type=True)
  File "/home/vee/.local/lib/python3.10/site-packages/srsly/msgpack/__init__.py", line 55, in packb
    return Packer(**kwargs).pack(o)
  File "srsly/msgpack/_packer.pyx", line 285, in srsly.msgpack._packer.Packer.pack
  File "srsly/msgpack/_packer.pyx", line 291, in srsly.msgpack._packer.Packer.pack
  File "srsly/msgpack/_packer.pyx", line 288, in srsly.msgpack._packer.Packer.pack
  File "srsly/msgpack/_packer.pyx", line 264, in srsly.msgpack._packer.Packer._pack
  File "srsly/msgpack/_packer.pyx", line 282, in srsly.msgpack._packer.Packer._pack
TypeError: can not serialize 'MyAttr' object

After some tests I think the issue is not caused by multiprocessing but by how do I assign the custom attribute. I created an issue regarding this : #10663
Thank you !

[ljvmiranda921]

Ok! Closing this discussion in favor of #10663