error in converting from conllu to spacy format

[toufiglu]

Hi,

I was converting a conllu file to spacy by python -m spacy convert

The conllu file does not seem to have problems at all, but I kept having: ValueError: not enough values to unpack (expected 10, got 1)

Traceback (most recent call last):
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/spacy/cli/convert.py", line 75, in convert_cli
    convert(
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/spacy/cli/convert.py", line 140, in convert
    db = DocBin(docs=docs, store_user_data=True)
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/spacy/tokens/_serialize.py", line 87, in __init__
    for doc in docs:
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/spacy/training/converters/conllu_to_docs.py", line 38, in conllu_to_docs
    for sent_doc in sent_docs:
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/spacy/training/converters/conllu_to_docs.py", line 74, in read_conllx
    set_ents = has_ner(input_data, ner_tag_pattern)
  File "/Users/yiminglu/opt/anaconda3/lib/python3.9/site-packages/spacy/training/converters/conllu_to_docs.py", line 58, in has_ner
    id_, word, lemma, pos, tag, morph, head, dep, _1, misc = parts
ValueError: not enough values to unpack (expected 10, got 1)

Just to give an example about the conllu file:

# sent_id = M2S30
# text = meme dangereux kima blida fiparout
1	meme	Même	ADV	ADV	_	4	advmod	_	lang=fr|ComposedUDPOS=ADV
2	dangereux	dangereux	ADJ	ADJ	_	4	amod	_	lang=fr|ComposedUDPOS=ADJ
3	kima	comme	ADV	ADV	_	4	advmod	_	lang=ar_dz|ComposedUDPOS=ADV
4	blida	Blida	PROPN	NPP	_	0	root	_	lang=ar_dz|ComposedUDPOS=PROPN
5-6	fiparout	_	_	_	_	_	_	_	lang=ar_dz
5	fi	à	ADP	P	AdpType=Prep	6	case	_	_
6	parout	partout	ADV	ADV	_	4	advmod	_	lang=ar_dz|ComposedUDPOS=ADV

I have got exactly ten items on each line. I verified that in python by the following codes. Nothing was printed. I assume that means every line has ten items.

file = open ("cleaned.train.NArabizi_treebank.conllu", "r")
lines = file.readlines()
for line in lines:
    if line.startswith("# text =") or line.startswith("# trad_fr = ") or line.startswith('\n') or line.startswith ("# sent_id ="):
        continue
    else:
        words = line.split("\t")
        if len(words) != 10:
            print (words)

I don't understand why bash keeps telling me that it only gets one item.

Thank you so much if any of you could help!

[adrianeboyd]

Please use code blocks instead of screenshots for code and text examples.

Can you paste or attach one sentence that fails with the converter? It could possibly be an encoding error, or another common error is that the format requires a blank line after the last sentence.

You can validate the format with the UD validation tools (https://github.com/universaldependencies/tools) just with validate.py --level 2 for the CoNLL-U format validation only.

[toufiglu]

Hi Adrianne,

Thanks so much for replying!! This is the first time I have asked a question on Github. Sorry about the formatting. I tried validation.py and I have got more than 1000 errors.

Format errors: 35
Metadata errors: 1159
Morpho errors: 68
Syntax errors: 31
*** FAILED *** with 1293 errors

Do I have to correct every single one of them to be able to convert? This is not my own dataset, so I am not sure to what extent I should work on rectifying it.

Thank you!

[adrianeboyd]

I think it's mainly the format errors that are the problem, but invalid UPOS could also cause problems for the spacy converter.

The validate script truncates the error output by default, but there should be details about the first few errors of each type, so check whether it's something systematic that's easy to fix. I suspect that the metadata errors aren't important in terms of use with the spacy converter, but it's hard to know without seeing the data or the errors. It might only be a matter of fixing some tabs or some typos in UPOS or FEATS values.

[toufiglu]

Thanks so much! I will try to fix them!