[Spacy nightly] Error in parsing docbin: "Error -3 while decompressing data: incorrect header check"

[esdu]

How to reproduce the behaviour

I exported some data from Prodigy with the prodigy data-to-spacy command (Prodigy 1.10.4, Spacy 2.3.2)

Then I tried to train an NER model with the latest transformer models from Spacy 3.0.0rc2, following the instructions here to generate a config and run "spacy train".

When I train the model, I get the following error. It looks like there's an issue parsing the spacy data. Is this because Spacy 2 and Spacy 3 docbin formats are incompatible?

(spacy3) ed@dev:~/projects/spacy3$ spacy train config.cfg --paths.train ./data_train.spacy --paths.dev ./data_eval.spacy
ℹ Using CPU

=========================== Initializing pipeline ===========================
Set up nlp object from config
Pipeline: ['transformer', 'ner']
Created vocabulary
Finished initializing nlp object
Traceback (most recent call last):
  File "/home/ed/anaconda3/envs/spacy3/bin/spacy", line 8, in <module>
    sys.exit(app())
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/spacy/cli/train.py", line 56, in train_cli
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/spacy/training/initialize.py", line 55, in init_nlp
    nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer)
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/spacy/language.py", line 1232, in initialize
    proc.initialize(get_examples, nlp=self, **p_settings)
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/spacy_transformers/pipeline_component.py", line 315, in initialize
    validate_get_examples(get_examples, "Transformer.initialize")
  File "spacy/training/example.pyx", line 62, in spacy.training.example.validate_get_examples
  File "spacy/training/example.pyx", line 41, in spacy.training.example.validate_examples
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/spacy/training/corpus.py", line 131, in __call__
    for real_eg in examples:
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/spacy/training/corpus.py", line 153, in make_examples
    for reference in reference_docs:
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/spacy/training/corpus.py", line 186, in read_docbin
    doc_bin = DocBin().from_disk(loc)
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/spacy/tokens/_serialize.py", line 243, in from_disk
    self.from_bytes(file_.read())
  File "/home/ed/anaconda3/envs/spacy3/lib/python3.7/site-packages/spacy/tokens/_serialize.py", line 201, in from_bytes
    msg = srsly.msgpack_loads(zlib.decompress(bytes_data))
zlib.error: Error -3 while decompressing data: incorrect header check

Your Environment

Spacy Nightly

• spaCy version: 3.0.0rc2
• Platform: Linux-5.0.0-38-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.9
• Pipelines: en_core_web_trf (3.0.0a0)

Spacy 2

• spaCy version: 2.3.2
• Platform: Linux-5.0.0-38-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.9

[adrianeboyd]

Hi, the names for the commands have ended up a bit confusing here, sorry. For prodigy v1, the data-to-spacy command outputs the spacy v2 training format, which is .json instead of .spacy (DocBin).

You need to convert the data-to-spacy output to .spacy for v3 using spacy v3's convert command:

spacy convert data.json .

This will save data.spacy in the current directory.

We should improve this error message so it's clearer what's going on when you try to load JSON as DocBin.

[esdu]

Thanks, that fixed it!