Begin automatic language detection support

[C-Loftus]

• Create a with_language_detection method on the fifo builder to greedily initialize the language detection models
• Create a send_lines_multilingual method (still playing with names for this) to automatically send lines while also setting the proper language.
• Put all language detection features under a feature flag
• Add tests
• Fix clippy and any formatting issues before review

[C-Loftus]

Fixed the example; thanks tait for your guidance on that. I am finding that I can detect between obvious differences like Russian vs English no issue but yeah as you mentioned on matrix, English vs Spanish does not work well. It appears that lingua needs either obvious differences like writing systems / diacritic characters (i.e. à ) or a larger context window.

I asked a question on the lingua repo here and we can see what they say. Worst case scenario, even if we implement just distinguishing between Hindu / English / Mandarin / Russian / Korean or other languages where the writing systems are distinct, that is still a win in my opinion, even though its not ideal.

pemistahl/lingua-rs#463

Will fix up clippy stuff and tidy stuff up once I (hopefully) hear back on that discussion and have a better sense of any potential optimization strategies.

[TTWNO]

Subscribed to the issue. We'll see what they say.

I've had difficulty with Lingua switching languages. I tried "Hello world and 你好，世界!" in one string and Lingua always marked the entire text as English (or Chinese if I swapped the order).

The extremely short context windows will hurt for sure, but I have heard that iOS isn't perfect either and you often have to swirch explicitly unless it's a unique writing system.

(Another thing I've been thinking about is how to support Spiel. If I get my crate working, maybe I can copy your work here and apply it there :)