"two-career" is being shown as "two career"

[arademaker]

Losing the original text? Is it the right thing to do?

((kind "wf")
   (form . "two")
   (lemmas "two")
   (tag . "ignore")
   (meta
    (sep . "-")
    (type . "num")))

we do have the sep for produze the original text. Question is:

1. is it easier to have the text tokenized in the buffer?
2. should we not distinguish between spaces and other separators?

Remember that default sep is space, so when a token doesn't have sep it is assumed sep=" ". See confusing explanation in https://github.com/own-pt/glosstag/blob/princeton/dtd/glosstag.dtd#L158-L161 for the glosstag corpus !!

[odanoburu]

Losing the original text?

not really losing, but not showing it properly indeed.

the proper tokenization (using sep as separator when available and a space as default) could be implemented, but then I'm not sure if any other corpus will have sep attributes to make it worthwhile… how is tokenization described by other tokenizers? could touch.py produce something akin to sep? would it be useful to do so?