feat(*): ODict V2

[Nickersoft]

ODict V2 is a major rewrite of the ODict compiler and format, rewriting all of the core logic in Rust and introducing several new features. Rewriting it Rust makes ODict more versatile, allowing it to be more easily called from other languages without having the CLI installed, as well as compatibility with Rust-based platforms like Tauri.

Breaking Changes

As ODict V2 is a complete rewrite of the compiler, it is no longer backwards-compatible with files compiled with ODict V1.

• The FlatBuffers schema has been replaced with the Rust-specific rkyv serialization library, which doesn't need external schema DSL files (+ faster lookups w/ built-in HashMap support)
• Snappy compression has been replaced with LZ4 (smaller files)
• bleve is now replaced with the faster tantivy library and uses the charabia tokenizer for better multilingual tokenization
• The built-in server was moved from the core library to the CLI with a multi-dictionary support and a new endpoint schema

What's New

• odict new command for initializing new ODXML files with the correct schema
• odict merge now has the ability to merge multiple dictionaries into a single base dictionary
• Custom tokenizers via the Tantivy SDK
• New Rust crate (still unnamed) and Rust SDK, complete with a more intuitive way to read/create/write dictionaries in code
• Node bindings via NAPI-RS, removing the need for having the CLI installed + odict service
• Native Python bindings via PyO3
• odict info command for printing out the top-level information of a file (format, name, etc.) in different formats (JSON, XML, etc.)

What's Deprecated

• The Go SDK no longer exists, as no good interop between Go + Rust really exists, but can be revisited if there is a strong community demand
• The hidden odict service command no longer exists

Remaining Todos for Core Library

• SQL dumping
• Built-in dictionary server