convert from conll2003 to ner-json loses tags

[LadyHangaku]

How to reproduce the behaviour

command: spacy convert filename.conll . --converter ner

File must be a in a CoNLL-2003 format.

Bug

When converting a file from CoNLL-2003 to an ner json file, a lot of the tags get lost in the process. Here are some snippets of the conll and ner files. In the first example, the tag gets lost, in the second one, with the same tags, the ner file maintained them:

CoNLL-2003:

Name _ O I-PER		
Last Name _ O I-PER

ner:

  "orth":"Name",
  "tag":"_",
  "ner":"O"
},
{
  "orth":"Last Name",
  "tag":"_",
  "ner":"O"

---

CoNLL-2003:

Another Name _ O I-PER
Another Last Name _ O I-PER

ner:

  "orth":"Another Name ",
  "tag":"_",
  "ner":"B-PER"
},
{
  "orth":"Another Last Name",
  "tag":"_",
  "ner":"L-PER"

My Environment

• spaCy version: 2.3.7
• Platform: Linux-4.15.0-144-generic-x86_64-with-debian-buster-sid
• Python version: 3.7.6

[adrianeboyd]

Hi, the problem is that this format doesn't support spaces in the tokens. The converter expects to be able to split the columns by whitespace and that the IOB annotation is in the same column in all lines after splitting, typically either always in the 2nd column or always in the 4th column depending on the dataset.

[adrianeboyd]

Hmm, I get the same results for both of those examples? What you probably need to check is that line.split() returns the same number of values for all the lines in your data.

[LadyHangaku]

Oh good idea! I just checked that and it seems like, beside the empty lines with only a "\n" in them, they all have len == 4... it's a mystery

[LadyHangaku]

Oh and the actual name used in the doc has no umlauts or some special chars.

[LadyHangaku]

Update:
I took a small part of the doc and made it an own file with round about 20 tokens. Then I used the -s flag and the conversion worked well. When I used -s on the whole original doc, again, it didn't work. So I suppose it has something to do with the document length?

[polm]

I really think that somewhere in your data there is a weird character that is messing up conversion, and it wasn't in the 20 token segment you took. Maybe try going over all 20 token segments or other natural chunks automatically and seeing if any of them cause an error/lose data to find the line at issue?

If you know you should never have _ as a tag you should be able to automate a check for that.