Exception E896 with using static vectors

[algo-sroy]

I am trying to load the pre trained static vectors using the init model command.
As an example I am trying to load the vectors from the en_core_web_md model.

I run the below init vectors command :

python -m spacy init vectors en ./static_vectors/en_core_web_md/vocab/strings.json ./output_static_vectors/ -n en_core_web_md.vectors

Output :

[2021-06-21 06:27:56,883] [INFO] Reading vectors from static_vectors/en_core_web_md/vocab/strings.json
701572it [00:01, 448542.86it/s]
[2021-06-21 06:27:58,495] [INFO] Loaded vectors from static_vectors/en_core_web_md/vocab/strings.json
✔ Successfully converted 701572 vectors
✔ Saved nlp object with vectors to output directory. You can now use
the path to it in your config as the 'vectors' setting in [initialize].

When i try to begin training with the added vectors:

python -m spacy train ./config_test.cfg --output ./output_test2/ --paths.train ./datasets/train_spacy_sample.spacy --paths.dev ./datasets/dev_spacy_sample.spacy --nlp.batch_size 20 --paths.vectors ./output_static_vectors/en_core_web_md.vectors

I get an exception E896.

ValueError: Buffer and memoryview are not contiguous in the same dimension.

RuntimeError: [E896] There was an error using the static vectors. Ensure that the vectors of the vocab are properly initialized, or set 'include_static_vectors' to False.

Kindly help.

[polm]

When using vectors from a pretrained model you don't have to convert them first. In your train step you can just use --paths.vectors en_core_web_md.

That said I am surprised your convert command didn't give you an error, I'll have to look at why that happens.

[adrianeboyd]

If you use -o accuracy, initialize.vectors is filled in, but it looks like otherwise the paths.vectors ends up as null as a result of the quickstart template, which is something we can fix since otherwise the paths settings are confusing.

Be aware that if you're using -o efficiency the default statistical models will not be configured to use the vectors even if you add them in initialize. (This is a difference from v2. In v2 they were detected automatically while training, but in v3, you have to enable it explicitly in the model parameters.)

[algo-sroy]

Is the paths.vectors ending up as null something that is going to be fixed in a further release? Is there any estimated timeline for that?

[adrianeboyd]

You can just edit your config if you want to use --paths.vectors instead:

[initialize]
vectors = ${paths.vectors}

The default configs from init config will be fixed in the next release.

[Norky101]

I am using Spacy v3.6 and I am getting the same error

RuntimeError: [E896] There was an error using the static vectors. Ensure that the vectors of the vocab are properly initialized, or set 'include_static_vectors' to False.

and my config already includes the

[initialize]
vectors = ${paths.vectors}

and

include_static_vectors

is not in my config?

Just putting it out there for future people on more modern spacy versions

[rmitsch]

Hi @Norky101, please open a new discussion and include more information about your use case (including the entire config). Thanks!