prob = -20.0 for all lemmas in en_core_web_lg

[ghost]

How to reproduce the behaviour

import spacy
nlp = spacy.load("en_core_web_lg")
[x.prob for x in nlp("the saliency of thawing ice")]

gives

[-20.0, -20.0, -20.0, -20.0, -20.0]

Also see #1590

Your Environment

• spaCy version: 2.3.2

• Platform: Linux-4.12.14-lp150.12.28-default-x86_64-with-glibc2.10

• Python version: 3.7.9

• ** en_core_web_lg version:

nlp.meta["version"]
'2.3.1'

[adrianeboyd]

Because they're no longer used as model features in v2, the probability tables aren't distributed with the models as of v2.3 to save space. The table is provided through spacy-lookups-data instead. See the section "Probability and cluster features" here for how to load them into an existing model: https://spacy.io/usage/v2-3#migrating

Note that the probability and cluster tables in spacy-lookups-data only include 1M tokens instead of 1.3M as in v2.2 models, so if you need to replicate the exact probability tables from v2.2, you'll want to export all the probabilities from a v2.2 model and import them in a custom v2.3 model.

[ghost]

@adrianeboyd Thank you for the detailed answer, I wasn't aware for this.