Unable to create Example for spancat training purposes [E983]

[jamnicki]

Hello,
I want to train SpanCategorizer with my data in Python API for query strategies (ex. based on prediction confidence) experiments of active learning methods

I have no idea what I'm doing wrong and I would be extremely grateful for help.

spaCy v.3.4.2, python v.3.8.13

nlp = spacy.blank("pl")
spancat = nlp.add_pipe("spancat")
for label in ["nam_liv_person", "nam_loc_gpe_city", "nam_loc_gpe_country"]:
    spancat.add_label(label)

docbin_train = DocBin().from_disk("./data/train.spacy")
docs_train = tuple(docbin_train.get_docs(nlp.vocab))

pprint(docs_train[0].to_json())

First Doc insight

{'ents': [{'end': 66, 'label': 'nam_loc_gpe_city', 'start': 61},
          {'end': 122, 'label': 'nam_loc_gpe_city', 'start': 116},
          {'end': 177, 'label': 'nam_loc_gpe_city', 'start': 168}],
 'spans': {'sc': [{'end': 66,
                   'kb_id': '',
                   'label': 'nam_loc_gpe_city',
                   'start': 61},
                  {'end': 122,
                   'kb_id': '',
                   'label': 'nam_loc_gpe_city',
                   'start': 116},
                  {'end': 177,
                   'kb_id': '',
                   'label': 'nam_loc_gpe_city',
                   'start': 168}]},
 'text': 'some public text',
 'tokens': [{'end': 7, 'id': 0, 'start': 0},
            {'end': 18, 'id': 1, 'start': 8},
                                 . . .
            {'end': 219, 'id': 35, 'start': 218}]}

def get_example_from_doc(doc):
    spans = list(doc.ents)
    group = SpanGroup(doc, name="sc", spans=spans)
    gold_ref = {"sc": group}
    example = Example.from_dict(doc, gold_ref)
    return example

def get_examples(docs):
    return [get_example_from_doc(doc) for doc in docs]

with nlp.select_pipes(enable="spancat"):
    optimizer = nlp.initialize()
    examples = get_examples(docs_train)
    nlp.update(examples, sgd=optimizer)

  ---------------------------------------------------------------------------
  KeyError                                  Traceback (most recent call last)
  Cell In [15], line 3
        1 # with nlp.select_pipes(enable=COMPONENT):
        2 optimizer = nlp.initialize()
  ----> 3 examples = [
        4     get_example_from_doc(doc)
        5     for doc in docs_train
        6 ]
        8 nlp.update(examples, sgd=optimizer)
  
  Cell In [15], line 4, in <listcomp>(.0)
        1 # with nlp.select_pipes(enable=COMPONENT):
        2 optimizer = nlp.initialize()
        3 examples = [
  ----> 4     get_example_from_doc(doc)
        5     for doc in docs_train
        6 ]
        8 nlp.update(examples, sgd=optimizer)
  
  Cell In [12], line 5, in get_example_from_doc(doc)
        3 group = SpanGroup(doc, name=SPANS_KEY, spans=spans)
        4 gold_ref = {SPANS_KEY: group}
  ----> 5 example = Example.from_dict(doc, gold_ref)
        6 return example
  
  File ~/miniconda3/envs/inzynierka/lib/python3.8/site-packages/spacy/training/example.pyx:111, in spacy.training.example.Example.from_dict()
  
  File ~/miniconda3/envs/inzynierka/lib/python3.8/site-packages/spacy/training/example.pyx:568, in spacy.training.example._fix_legacy_dict_data()
  
  KeyError: "[E983] Invalid key(s) for 'token_annotation': sc. Available keys: {'ORTH', 'pos', 'MORPH', 'HEAD', 'heads', 'morphs', 'SPACY', 'SENT_START', 'LEMMA', 'deps', 'TAG', 'POS', 'DEP', 'lemmas', 'sent_starts', 'spaces', 'tags', 'words'}"

[polm]

Sorry you're having trouble with this.

Normally we apprecitate a lot of detail with questions, but in this case you have provided too much and it makes your question hard to follow, partly just because it's a lot to read. In this case, it would have helped if you narrowed things down to just the error you're getting and the code that caused it.

Looking at the specific error you're getting, you're passing invalid values to Example.from_dict. I'm not sure how you came up with your get_example_from_doc code, but it won't work - the format of data you're passing is not what that function is designed for. Instead what you should do is just set your spans on a Doc and save a DocBin. Assuming you have entities set in doc.ents but want to set them in spans, you can do that like this:

db = DocBin()

for doc in train_docs:
    doc.spans["sc"] = list(doc.ents)
    db.add(doc)

db.to_disk(... output file ...)

You do not need to use doc.to_json or doc.to_dict. Do not set scores on the training data, it won't do anything. I would also recommend you use the config-based training instead of writing your own training loop to verify that training works at all first, and then work on a custom training loop for your active learning.

[jamnicki]

I created DocBin following your example above. Trained a model on it using cli with results presented below:

config_sm.cfg file content

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "pl"
pipeline = ["tok2vec","spancat"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "sc"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat.suggester]
@misc = "spacy.ngram_suggester.v1"
sizes = [1,2,3]

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

python -m spacy debug data config/spacy/config_sm.cfg --system.seed 42 --paths.train ./data/inzynierka-kpwr-train-3-full.spacy --paths.dev ./data/inzynierka-kpwr-test-3-full.spacy 

debug data output

/home/jjamnicki/miniconda3/envs/bachelor_thesis_project/lib/python3.8/site-packages/torch/cuda/__init__.py:497: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
/home/jjamnicki/miniconda3/envs/bachelor_thesis_project/lib/python3.8/site-packages/torch/cuda/__init__.py:529: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() if nvml_count < 0 else nvml_count

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: pl
Training pipeline: tok2vec, spancat
3456 training docs
1136 evaluation docs
⚠ 12 training examples also in evaluation data

============================== Vocab & Vectors ==============================
ℹ 69452 total word(s) in the data (20724 unique)
ℹ No word vectors present in the package

============================ Span Categorization ============================

Spans Key   Labels                        
---------   ------------------------------
sc          {'nam_loc_gpe_country', 'nam_liv_person', 'nam_loc_gpe_city'}

ℹ Span characteristics for spans_key 'sc'
ℹ SD = Span Distinctiveness, BD = Boundary Distinctiveness

Span Type             Length     SD     BD      N
-------------------   ------   ----   ----   ----
nam_liv_person          1.50   2.49   1.15   2911
nam_loc_gpe_city        1.10   3.62   1.32   1342
nam_loc_gpe_country     1.11   3.90   1.40    992
-------------------   ------   ----   ----   ----
Wgt. Average            1.33   3.04   1.24      -

ℹ Over 90% of spans have lengths of 1 -- 2 (min=1, max=7). The most
common span lengths are: 1 (64.96%), 2 (29.49%). If you are using the n-gram
suggester, note that omitting infrequent n-gram lengths can greatly improve
speed and memory usage.
✔ Spans are distinct from the rest of the corpus
✔ Boundary tokens are distinct from the rest of the corpus
✔ Good amount of examples for all labels
✔ Examples without ocurrences available for all labels

================================== Summary ==================================
✔ 6 checks passed
⚠ 1 warning

python -m spacy train --verbose config/spacy/config_sm.cfg --output ./models/cli-trained-spancat-kpwr-full.spacy --system.seed 42 --paths.train ./data/inzynierka-kpwr-train-3-full.spacy --paths.dev ./data/inzynierka-kpwr-test-3-full.spacy

train output

/home/jjamnicki/miniconda3/envs/bachelor_thesis_project/lib/python3.8/site-packages/torch/cuda/__init__.py:497: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
/home/jjamnicki/miniconda3/envs/bachelor_thesis_project/lib/python3.8/site-packages/torch/cuda/__init__.py:529: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() if nvml_count < 0 else nvml_count
[2022-11-15 17:15:20,054] [DEBUG] Config overrides from CLI: ['system.seed', 'paths.train', 'paths.dev']
✔ Created output directory:
models/cli-trained-spancat-kpwr-full.spacy
ℹ Saving to output directory:
models/cli-trained-spancat-kpwr-full.spacy
ℹ Using CPU

=========================== Initializing pipeline ===========================
[2022-11-15 17:15:20,422] [INFO] Set up nlp object from config
[2022-11-15 17:15:20,431] [DEBUG] Loading corpus from path: data/inzynierka-kpwr-test-3-full.spacy
[2022-11-15 17:15:20,432] [DEBUG] Loading corpus from path: data/inzynierka-kpwr-train-3-full.spacy
[2022-11-15 17:15:20,432] [INFO] Pipeline: ['tok2vec', 'spancat']
[2022-11-15 17:15:20,436] [INFO] Created vocabulary
[2022-11-15 17:15:20,436] [INFO] Finished initializing nlp object
[2022-11-15 17:15:24,345] [INFO] Initialized pipeline components: ['tok2vec', 'spancat']
✔ Initialized pipeline

============================= Training pipeline =============================
[2022-11-15 17:15:24,353] [DEBUG] Loading corpus from path: data/inzynierka-kpwr-test-3-full.spacy
[2022-11-15 17:15:24,354] [DEBUG] Loading corpus from path: data/inzynierka-kpwr-train-3-full.spacy
ℹ Pipeline: ['tok2vec', 'spancat']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  ------------  ----------  ----------  ----------  ------
  0       0         10.44        246.39        1.45        0.75       26.74    0.01
  0     200         23.15       1734.84       48.75       55.18       43.66    0.49
  0     400          2.98       1151.66       50.87       72.30       39.24    0.51
  1     600          2.24       1249.81       57.71       71.15       48.54    0.58
  1     800          3.51       1030.63       63.53       82.70       51.58    0.64
  2    1000          3.66        916.07       70.36       77.50       64.43    0.70
  3    1200          4.12        797.97       70.81       77.67       65.06    0.71
  4    1400          3.48        664.85       71.25       81.52       63.28    0.71
  5    1600          3.43        612.71       71.55       80.44       64.43    0.72
  6    1800          3.83        624.41       71.59       78.01       66.15    0.72
  8    2000          3.20        601.48       72.79       76.93       69.08    0.73
 11    2200          3.46        667.97       73.30       82.54       65.92    0.73
 13    2400          2.99        717.81       73.24       78.23       68.85    0.73
 16    2600          2.27        670.45       73.29       79.34       68.10    0.73
 19    2800          1.82        622.43       73.28       79.70       67.81    0.73
 22    3000          1.68        617.95       72.58       76.89       68.73    0.73
 25    3200          3.24        575.17       71.99       77.37       67.30    0.72
 27    3400          2.42        479.11       73.92       80.11       68.62    0.74
 30    3600          2.36        416.87       74.14       82.70       67.18    0.74
 33    3800          1.80        396.29       73.81       77.16       70.74    0.74
 36    4000          1.66        366.62       73.78       76.37       71.37    0.74
 39    4200          1.86        397.35       73.59       81.52       67.07    0.74
 41    4400          1.37        384.79       72.77       82.38       65.17    0.73
 44    4600          1.41        377.31       75.16       81.92       69.42    0.75
 47    4800          1.48        378.25       73.63       80.95       67.53    0.74
 50    5000          1.37        357.54       73.50       79.06       68.67    0.74
 53    5200          1.53        368.40       73.82       77.32       70.63    0.74
 56    5400          1.77        355.94       73.29       80.22       67.47    0.73
 58    5600          1.43        351.07       72.72       75.60       70.05    0.73
 61    5800          1.20        341.93       72.59       80.17       66.32    0.73
 64    6000          1.25        331.24       73.50       79.60       68.27    0.74
 67    6200          0.49        321.52       72.51       79.48       66.67    0.73
✔ Saved pipeline to output directory
models/cli-trained-spancat-kpwr-full.spacy/model-last

It think it is all right.

I also implemented new loop iterating over examples as you wrote using Corpus class.
Take a look at my code if you would be so kind and willing to help because I still want to stick with Examples 😃

Again thank you for the response, you helped a lot!

[polm]

Thanks for using the config-based training to check everything, that make it clear your data is OK.

Looking at the linked code, the way you create Examples using Corpus looks fine. Are you having issues with it? I'm not clear what you still need help with.

[jamnicki]

Actually, I found an article on medium about custom spancat training on examples created from scratch

The reason why I wanted more help was occuring warning about no span to render im displacy but I am almost sure it is because of treshold. Spans list under my key exists but it is simply empty.

Thank you!

[polm]

OK, thanks for letting us know what's up. If it's not as simple as a threshold issue, feel free to follow up here or in a new thread.

[jamnicki]

Hello, @polm Python API newbie strikes back... I started 2 related discussions: #11835 and #11829.
My highest priority issue is to achieve satisfactory results from Python API trained spancat.