Japanese transformers-based model

[hiroshi-matsuda-rit]

After discussions with @polm, @tamuhey, and @KoichiYasuoka in the spaCy Japanese community, we came to the conclusion that we would propose the publication of a transformers-based Japanese analysis model:

• Base model
  • https://huggingface.co/cl-tohoku/bert-base-japanese-char-v2
• Tokenizer settings
  • Use the basic pretokenizer instead of mecab to remove fugashi and unidic-lite dependencies
  • This replacement of pretokenizer is very easy by adding a [components.transformer.model.tokenizer_config] section in config file
  • Training configuration example: https://github.com/megagonlabs/UD_Japanese-GSD/blob/master/spacy/config/ja_gsd_bert_char_v2_basic.cfg#L96

With these settings, the accuracy of spaCy v3 Japanese model will be greatly improved.
Although not as good as the fully pretokenized BERT model, the character-based BERT model can eliminate memory consumption, pretokenizer overhead, and meny-to-many token alignments.
https://github.com/megagonlabs/UD_Japanese-GSD/blob/master/leader_board.md
https://docs.google.com/spreadsheets/d/1D1MvywJYCSVCHaL8p9MRjwL7CHmWQFLIH--kqW8DSmQ/edit#gid=743830749

[hiroshi-matsuda-rit]

Related issues: #8677 #8977

[adrianeboyd]

Thanks, we'll try it out!