Spyc Convert Does not detect document boundaries on conll203

[maximilianreimer]

How to reproduce the behaviour

When using spcy convert -c ner inputfile.txt . where inputfile:

-DOCSTART- -X- -X- O

SOCCER NN I-NP O
- : O O
JAPAN NNP I-NP B-LOC
GET VB I-VP O
LUCKY NNP I-NP O
WIN NNP I-NP O
, , O O
CHINA NNP I-NP B-LOC
IN IN I-PP O
SURPRISE DT I-NP O
DEFEAT NN I-NP O
. . O O

...

-DOCSTART- -X- -X- O is not used to determine document delimiters instead each sentence is parsed into one document.

Info about spaCy

• spaCy version: 3.0.6
• Platform: Linux-5.11.0-1025-aws-x86_64-with-debian-bullseye-sid
• Python version: 3.7.11
• Pipelines: nb_core_news_lg (3.0.0), en_core_web_trf (3.0.0)

[adrianeboyd]

The document boundary line that the converter uses is:

-DOCSTART- -X- O O

Edited to add:

See the notes and sample data in the docs: https://spacy.io/api/cli#converters

[maximilianreimer]

I did not see the difference. Which datasets follow this notation? I found a variant of Conll2003 and Conll++ which follow the delimiter I listed. And is there a way of configuring that?

In general: is there a way of loading IOB files in code?

[adrianeboyd]

I'm pretty sure that this was the delimiter used in the original CoNLL 2003 dataset, but it's possible it wasn't consistent or others have been introduced since then. In any case, this is the delimiter that this particular converter has used for many years. It isn't currently configurable, but it shouldn't be too hard to modify your data (or modify the converter, if you'd prefer).

As long as you can set your annotation in a Doc, you can save it in a DocBin for training. If you have this kind of format, it's probably easiest to use Doc(nlp.vocab, words=words, ents=ent_iob_tags) for each document. You may also want to add the sentence breaks with sent_starts=[True, False, False, ...], which is also as a list of per-token values like words and ents.