Appending morphologizer to Japanese pipeline

[hiroshi-matsuda-rit]

I found the morphologizer of spaCy v3 works fine with Japanese pipeline.
We can find significant improvement for UPOS but no obvious degrades in any other metrics.
Please try ja_gsd-3.1.1.tar.gz below.
https://github.com/megagonlabs/UD_Japanese-GSD/releases/tag/r2.8-NE
https://github.com/megagonlabs/UD_Japanese-GSD/blob/master/leader_board.md

We're also evaluating Japanese pipelines with various transformers settings including UD centric ELECTRA pretrained model.
After discussing about pros and cons of these model settings in our gitter room, we'd like to send the best one here.
https://gitter.im/spaCy_ja/spacy-transformers?utm_source=share-link&utm_medium=link&utm_campaign=share-link

Thanks,

[polm]

Thanks for the report! It's great to hear the morphologizer can improve the quality of POS tags. I guess this means we can get rid of the old hacky tag conversion too.

Looking forward the the ELECTRA model! Can you clarify what's "proprietary" about it?

[hiroshi-matsuda-rit]

Sure. We can remove current bi-gram POS tagging rules.
https://github.com/explosion/spaCy/blob/master/spacy/lang/ja/__init__.py#L57

We're also thinking about adding an extra user_data field which contains normalized_form of SudachiPy, like reading_forms.
https://github.com/explosion/spaCy/blob/master/spacy/lang/ja/__init__.py#L72

Can you clarify what's "proprietary" about it?

Sorry. That's not a good word.
I'm developing the ELECTRA pretraining models by using NINJAL's NWJC text dataset, which contains 630 million sentences, and SudachiPy mode A + worpiece for tokenization.
Some of these ELECTRA models will be released within a few weeks under the CC BY-SA 4.0 license.

[adrianeboyd]

Is it useful to have the hacky tag conversion as a backoff for cases where there isn't a morphologizer? I don't think it's a problem to leave it in the tokenizer and let the morphologizer overwrite those values if it's present.

[hiroshi-matsuda-rit]

I have confirmed that there is no difference in the output of morphologizer depending on whether token.pos is set during tokenization. I think it's a good idea to leave the rule-based POS tagger.

[adrianeboyd]

It's simple to add the morphologizer to the internal config for Japanese, so we can do that for the next round of model training.