Support for locale-specific indexes

[kaedroho]

See previous discussion on Wagtail: wagtail/wagtail#1177

We need:

• A way to specify the language of an index, which will enable the language specific-features of the backend (ES/OS only as database backends can support multiple languages in a single index)
• A way to specify the language(s) of a piece of content so it can be indexed in the correct index
• A way to specify the language of the query, which will enable the language specific-features of the backend

Needs to be agnostic of how the developer has structured their multi-lingual content (separate fields, separate records). and not tied in to any particular content internationalisation library (django-modeltranslation, Wagtail, etc).

[Nightwing-77]

hey! so in this we could go ahead and index all of your content in multiple indexes, with each index having a configuration for a different language. we could select that specific indexes when searching. or we could add another field in the index for language which we could filter while searching. also i had some question about language mixing, what if user uses language mixed text (i'm just thinking)

i could think the option of adding the language index would be a better option than having seperate indexes for each language!

[kaedroho]

i could think the option of adding the language index would be a better option than having seperate indexes for each language!

Could you elaborate on any reasons why you think a single index would be better?

I think multiple indexes would be best for Elasticsearch/OpenSearch because it would allow you to configure different analyzers for different languages (for example, English, Turkish, and Chinese all require different ways of extracting words from text). And if there are any tokens that are used in multiple languages but have different meanings or frequencies of use, this could make the term weights less accurate.

For the database, we could use a single table. See @BertrandBordage's comment here wagtail/wagtail#1177 (comment)

[Nightwing-77]

i could think the option of adding the language index would be a better option than having seperate indexes for each language!

Could you elaborate on any reasons why you think a single index would be better?

I think multiple indexes would be best for Elasticsearch/OpenSearch because it would allow you to configure different analyzers for different languages (for example, English, Turkish, and Chinese all require different ways of extracting words from text). And if there are any tokens that are used in multiple languages but have different meanings or frequencies of use, this could make the term weights less accurate.

For the database, we could use a single table. See @BertrandBordage's comment here wagtail/wagtail#1177 (comment)

it might be a subjective opinion but multi lingual search i feel has it's own pros as well ! like i don't wanna specify which language i wanna search every time and like maybe i'm overthinking it but it's like enforcing a extra rule on user for a problem that might not even exist for them!! i'm trying to think about it from users perspective . also language specification would be a strict requirement also the user know which language the text that they're searching for . it's just too much extra work for the user without any strong benefit (purely my personal opinion)

[Nightwing-77]

i'll start working on this! will add locale field to IndexEntry and then filter by that field to match the current locale ! will use wagtail/wagtail#1177 (comment) as a reference