Problem with debug data

[tohidmonjazeb]

How to reproduce the behaviour

python -m spacy debug data ./config.cfg

Your Environment

• Operating System: windows 10
• Python Version Used: 3.8.5
• spaCy Version Used: 3.1.3
• Environment Information:
• I have made train.spacy and dev.spacy files then I ran "python -m spacy debug data ./config.cfg" and nothing happened
  I mean that it just shows :

2021-11-27 13:49:32.956042: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-11-27 13:49:32.956288: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

============================ Data file validation ============================

And nothing else even when I wait for 1 hour.
Could you please help me with this?

[polm]

spacy debug data loads the data into memory by default. After one hour I'd expect you to have run out of memory, but maybe something weird is going on. How big is the data you're passing to debug data, in file size, number of docs, number of words? Can you share your config and the output of spacy info?

[polm]

Thanks! It looks like dep annotations have a problem and your data is full of cycles. debug data shouldn't hang in this situation, but I don't think we've tested it much with invalid data, so that's a bug we need to fix, but your data isn't usable in its current state.

I'm not sure how you generated your annotations, but you seem to have many situations where two words, A and B, will be each other's heads, which is incorrect. The dependency relations should form an upside-down tree with one root.

[polm]

Actually, one small note - while you do have the situation I described where two nodes form a cycle often, it looks like even more often you have a situation where a word is its own head. That should only be the case for the root in the sentence (so just one word).

[tohidmonjazeb]

Actually I got this data set from https://github.com/UniversalDependencies/UD_Persian-PerDT
after examining it I faced with many wrong annotations, so I tried to fix them but it seems that I should not use this data at all.
Anyway, I'm so thankful for your guidance and responsibility .

[adrianeboyd]

We provide converters for the UD CoNLL-U format. Since this UD corpus contains multi-word tokens, which spacy Docs don't support, I'd usually recommend converting it like this (every ten sentences in a paragraph, merge multi-word tokens to tokens that spacy supports):

python -m spacy convert -n 10 -T fa_perdt-ud-train.conllu .

[tohidmonjazeb]

Actually I didn't use spacy's converter because this data contains many null values and also some wrong annotations with heads.
so I preprocessed it manually and made Doc objects and added them to DocBin and then made spacy files from them.