Handling of text metadata in mSUD annotated treebanks

[bguil]

When a sentence is annotated in mSUD, there are two different text information to handle.
For instance in sentence AN14_06-06 in the mSUD Ika treebank, there are two mSUD tokens: a- and gwasi.
We then have "text" representation:

• morph level: a- gwasi
• word level: agwasi

We need to decide the way to encode this consistently in mSUD treebanks.

In our current mSUD data we have a very inconsistent state:

lang	morph level text	word level
Beja	inconsistent phonetic_text / text	inconsistent
Gbaya	text	phonetic_text
Pesh	morphemic_text	text
Tuwari	text	not available
Ika & Bokota (in ArboratorGrew)	morphemic_text	text_ortho & text

I suggest to choose the "Pesh" line because the name morphemic_text is more explicit and less confusing than phonetic_text

Note that this would also need to adapt ArboratorGrew to take into account the fact that mSUD is special in this respect:

• in UD and SUD: text is the contatenation of token's FORM (taking into account MWT tokens and SpaceAfter)
• in mSUD: text is not the contatenation...

Note also that in Chinese, Taigi and Teochew treebanks, morphs do not contains the - symbol, only one "text" field is needed.

[Martine-Vanhove]

phonetic_text is a remnant of a previous label in ELAN software for Beja. There is no problem to change phonetic_text to morphemic_text for this language. Some files are indeed inconsistent for the text line and all morph breaks (- and =) should be erased and the morphs should be concatenated with the previous or following morph. Can this be done automatically?

[bguil]

In Beja and other languages where clitics are explicitly marked with =, we have in fact one more "text" level!
For the sentence BEJ_MV_NARR_09_jewel_52-52:

• 3 tokens in mSUD:
  • giːg
  • -i
  • =t
• 2 tokens in SUD or UD:
  • giːgi
  • =t

For this sentence, in the 2.15 data:

• giːg -i =t → phonetic_text in mSUD data
• giːgi =t → text in UD/SUD treebank (This is required by the UD validator: text must be the concatenation of the 2 tokens giːgi and =t)
• giːgit → text in mSUD treebank

[sylvainkahane]

For corpora with several tokenisations, we need a constant name for each one. I don't know which one, but one. It could be:

• morph-based_text or morphemic_text or m_text
• word-based_text or lexeme-based_text or w_text or l_text

When we have two notions of "word" (with or without clitics), I don't know what would be the best terminology: short vs long words? any proposition?

text can change according to the tokenisations chosen for the annotation. While m_text and w_text will be constant, whatever the version of the treebank.

[natalia-caceres-arandia]

text can change according to the tokenisations chosen for the annotation. While m_text and w_text will be constant, whatever the version of the treebank.

This is the kind of solution I would prefer : multiple x_text lines allowing for different representations of the text, and the text field corresponding to the segmentation that indicated by the nWord field.

Why?
Currently, in mSUD, text is automatically regenerated when we use the split token function, and so the original text information is replaced (but it would always be preserved if we have a w_text field containing that original information).

[bguil]

Thanks for your comments.

I agree with the idea to have stable "x_text" fields, whatever is the tokenisation level and the annotation framework (UD vs SUD). It would also be nice to keep these "x_text" stable among treebanks and languages!

As @sylvainkahane proposed the field named text would be the contatenation of token's FORM (taking into account MWT tokens and SpaceAfter) and so we can have different text for the same sentence in mSUD or SUD:

• This is was is expected but the UD validator
• This is also how ArboratorGrew handles it: after a change of tokenisation or after the edition for the FORM of a token: the field text is rebuilt to follow this rule.

---

Coming back to Beja's example, we can set:

# sent_id = BEJ_MV_NARR_09_jewel_52-52
# m_text = giːg -i =t
# c_text = giːgi =t
# w_text = giːgit

With c_text for "clitic level text", but maybe we do not need to consider this level and we can skip it?

---

Another example with a Pesh sentence:

We would have:

• m_text = pa a- kas- teʃ teʔk -u -h =wa
• w_text = pa akasteʃteʔkuh wa
• for text:
  • in mSUD: text = pa a- kas- teʃ teʔk -u -h =wa
  • in SUD: text = pa akasteʃ teʔkuh =wa (SUD is obtained from mSUD by merging affixes)
  • in wSUD: text = pa akasteʃteʔkuh wa (wSUD is obtained from mSUD by following features nWord and Word)

[sylvainkahane]

I just have an hesitation concerning the term "word". I think that in its most common sense, clitics are analyzed as words (and clitics are orthographic words in English or in French). I don't have a perfect solution to propose and I don't think that anyone would. It why I proposed "short word" and "long word". Which will give us sw_text and lw_text.

There is also the problem of portmanteau words, like "au" in French, which merges "à" and "le". Here too we need two x_text and I don't know how to call them. Maybe word with spaces, like Fr. "parce que", are a problem too. Do we want to have an x_text indicating that parce_que can be one token? Do we need this?

[bguil]

If I understand correctly the last proposal we would have:

# sent_id = BEJ_MV_NARR_09_jewel_52-52
# m_text = giːg -i =t
# sw_text = giːgi =t
# lw_text = giːgit

For the other remark, the conllu format already implements a mechanism for portmanteau words with multi-word tokens. We don't want to introduce another one.
For words with space, I don't see a need for tokens like parce_que, the infomation is available (fixed in UD, unk+Idiom in SUD).

[sylvainkahane]

I change my mind about the notations after discussing the data from Gbaya. I think that "long word" are in fact phonological words (like Fr. du, au, l'olive), while "small words" are syntactic words (= inflected lexemes). I propose:

# m_text = giːg -i =t
# s_text = giːgi =t
# p_text = giːgit

with m for morph, s for syntax, p for phonology.