spacy debug data doesn't show where it saw an error

[dmoti]

I'm training a NER model on Hebrew text.

After an error in training:

ValueError("[E024] Could not find an optimal move to supervise the parser.
Usually, this means that the model can't be updated in a way that's valid and
satisfies the correct annotations specified in the GoldParse. For example, are
all labels added to the model? If you're training a named entity recognizer,
also make sure that none of your annotated entity spans have leading or trailing
whitespace or punctuation. You can also use the `debug data` command to validate
your JSON-formatted training data. For details, run:\npython -m spacy debug data
--help")

I tried to run spacy debug data (with verbose option), I'm getting an error report (6 invalid whitespace entity spans) but I don't see where the error are, I scanned the data myself to locate these spaces but didn't find them

(venv) (base) mdaniel@newai-gpu01:/data/home/mdaniel/work/ner_demo$ python -m spacy project run debug

=================================== debug ===================================
Running command: /data/home/mdaniel/work/ner_demo/venv/bin/python -m spacy debug data configs/config.cfg --code ./heb_tokenizer.py --verbose --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy
/data/home/mdaniel/work/ner_demo
/data/home/mdaniel/work/ner_demo/venv/lib/python3.8/site-packages/hebrew_tokenizer/tokenizer.py:121: FutureWarning: Possible nested set at position 843
  self.scanner = re.compile(

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: he
Training pipeline: tok2vec, ner
18798 training docs
2089 evaluation docs
✔ No overlap between training and evaluation data

============================== Vocab & Vectors ==============================
ℹ 400691 total word(s) in the data (55392 unique)
10 most common words: 'של' (10646), 'על' (4181), 'את' (3905), 'או' (3516), 'עם'
(2879), 'יש' (1582), 'יותר' (1554), 'טיפול' (1433), 'of' (1361), 'גם' (1356)
ℹ No word vectors present in the package

========================== Named Entity Recognition ==========================
ℹ 5 label(s)
0 missing value(s) (tokens with '-' label)
Labels in train data: 'CONDITION', 'ORGAN', 'HORMON', 'PROCEDURE', 'MEDICINE'
✘ 6 invalid whitespace entity spans
✔ Good amount of examples for all labels
✔ Examples without occurrences available for all labels
Entity spans consisting of or starting/ending with whitespace characters are
considered invalid.

================================== Summary ==================================
✔ 5 checks passed
✘ 1 error

I'm using spacy version 3.2 with python 3.8.8

[dmoti]

Found a way to clean my data:
#3558

[shahidbasheer]

I did this hack and it works perfect.

Edit this file and save these annotations in text file
prodigy\lib\python3.8\site-packages\spacy\cli\debug_data.py

if label.startswith(("B-", "U-", "L-")) and doc[i].is_space:
                    # "Illegal" whitespace entity
                    f = open('whitespace-annotations .txt', 'a')
                    f.write(doc.text)
                    f.write('\n==========================================\n')
                    f.close()

2. export prodigy to jsonl with db-out
3. find these annotations in jsonl
4. delete the line and import it back in prodigy with db-input