FIX: Normalize languages within Discourse AI translator #194
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nattsw
force-pushed
the
normalize-language
branch
7 times, most recently
from
b5da794 to
07eabc3
Compare
The criteria to allow a post to be translated is if the user's locale is different from the site's default locale. For each translator (e.g. Google, Amazon, Azure), we maintain a list of mappings `SUPPORTED_LANG_MAPPING` for each locale. On top of that, th is mapping is also used to define what language the post should be translated to. When using most translators available in the plugin, "en_GB" forums will translate text to "en" due to these mappings. This is not a problem for Discourse AI provider, since there are no restrictions on language. This PR introduces a site setting `normalize_language_variants_map` so that admins can properly indicate if A. A. They want `"pt" == "pt_BR" == "pt_PT"` or `"zh" == "zh_CN" == "zh_TW"` or B. They would prefer that `"en"` is not the same as `"en_GB"`.
nattsw
force-pushed
the
normalize-language
branch
from
07eabc3 to
6eaac39
Compare
Member
|
My only concern here is that this problem is being offloaded to the admin, should the plugin just handle all of this transparently? Meaning use maps for providers but pass on raw information to Discourse AI? |
Contributor
Author
|
My original implementation was for the site setting to be a simple toggle, with normalize by default. We can try something simpler for now without a site setting, which is to have Discourse AI just normalize by default. |
nattsw
changed the title
SamSaffron
approved these changes
Jan 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the user's locale is different from the post's detected locale, a post can be translated, and the 🌐 icon will appear.
For each translator (e.g. Google, Amazon, Azure), we maintain a list of mappings
SUPPORTED_LANG_MAPPINGfor each locale. On top of that, the mapping is also used to define what language the post should be translated to. When using most translators, "en_GB" forums will translate text to "en" due to these mappings. However, this is not a problem for Discourse AI provider -- since there are no restrictions on language, every post is translatable to any (regionalised) locale.This PR introduces a site settingnormalize_language_variantsso that admins can properly indicate if (example locale)They want"ar" == "ar_SA" == "ar_MA"or;They would prefer that"ar_SA"is not the same as"ar".This PR normalizes locales within Discourse AI translator, so that it sees "ar" to be the same as "ar_SA".