docs

vibrato/docs

Here provides several documents for advanced usages.

• compile.md describes how to compile system dictionaries from language resources.
• map.md describes how to achieve faster tokenization through cache-efficient id mapping.
• train.md describes how to train dictionaries from your own corpus.
• small-dic.md describes how to generate smaller system dictionaries.
• benchmark describes how to benchmark the tokenization speed of Vibrato.