[Bug]: Using infinity as the DOC_ENGINE, Chinese content cannot be retrieved

[asiroliu]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

RAGFlow workspace code commit ID

279b01a

RAGFlow image version

x

Other environment information

infiniflow/infinity:v0.7.0-dev2
RAGFlow build by source code. commit id: 279b01a02

Actual behavior

Using infinity as the DOC_ENGINE, Chinese content cannot be retrieved

Expected behavior

No response

Steps to reproduce

Using infinity as the DOC_ENGINE to create a Chinese knowledge base, no information can be retrieved whether in retrieval, chat, or agent modes

Additional information

infiniflow=> select content from ragflow_df72e7dc032011f198f48f655838f485_ed351cec033811f1975c0fda319450ca;
                                                                 content
------------------------------------------------------------------------------------------------------------------------------------------
                                                                                                                                         +
 先帝创业未半而中道崩殂，今天下三分，益州疲弊，此诚危急存亡之秋也。\r
                                                                                                                                         +
 然侍卫之臣不懈于内，忠志之士忘身于外者，盖追先帝之殊遇，欲报之于陛下也。\r
                                                                                                                                         +
 诚宜开张圣听，以光先帝遗德，恢弘志士之气，不宜妄自菲薄，引喻失义，以塞忠谏之路也。\r
                                                                                                                                         +
 #\r宫中府中，俱为一体，陟罚臧否，不宜异同。\r若有作奸犯科及为忠善者，宜付有司论其刑赏，以昭陛下平明之理，不宜偏私，使内外异法也。\r
                                                                                                                                         +
 #\r侍中、侍郎郭攸之、费祎、董允等，此皆良实，志虑忠纯，是以先帝简拔以遗陛下。\r
                                                                                                                                         +
 愚以为宫中之事，事无大小，悉以咨之，然后施行，必能裨补阙漏，有所广益。\r
                                                                                                                                         +
 #\r将军向宠，性行淑均，晓畅军事，试用于昔日，先帝称之曰能，是以众议举宠为督。\r
                                                                                                                                         +
 愚以为营中之事，悉以咨之，必能使行阵和睦，优劣得所。\r
                                                                                                                                         +
 #\r亲贤臣，远小人，此先汉所以兴隆也；亲小人，远贤臣，此后汉所以倾颓也。\r
                                                                                                                                         +
 先帝在时，每与臣论此事，未尝不叹息痛恨于桓、灵也。侍中、尚书、长史、参军，此悉贞良死节之臣，愿陛下亲之信之，则汉室之隆，可计日而待也。\r
                                                                                                                                         +
 #\r臣本布衣，躬耕于南阳，苟全性命于乱世，不求闻达于诸侯。\r
                                                                                                                                         +
 先帝不以臣卑鄙，猥自枉屈，三顾臣于草庐之中，咨臣以当世之事，由是感激，遂许先帝以驱驰。\r
                                                                                                                                         +
 后值倾覆，受任于败军之际，奉命于危难之间，尔来二十有一年矣。\r
                                                                                                                                         +
 #\r先帝知臣谨慎，故临崩寄臣以大事也。\r受命以来，夙夜忧叹，恐托付不效，以伤先帝之明，故五月渡泸，深入不毛。\r
                                                                                                                                         +
 今南方已定，兵甲已足，当奖率三军，北定中原，庶竭驽钝，攘除奸凶，兴复汉室，还于旧都。\r
                                                                                                                                         +
 此臣所以报先帝而忠陛下之职分也。至于斟酌损益，进尽忠言，则攸之、祎、允之任也。\r
                                                                                                                                         +
 #\r愿陛下托臣以讨贼兴复之效，不效，则治臣之罪，以告先帝之灵。\r
                                                                                                                                         +
 若无兴德之言，则责攸之、祎、允等之慢，以彰其咎；陛下亦宜自谋，以咨诹善道，察纳雅言，深追先帝遗诏，臣不胜受恩感激。\r
                                                                                                                                         +
 #\r今当远离，临表涕零，不知所言。
(19 rows)

infiniflow=> select content from ragflow_df72e7dc032011f198f48f655838f485_ed351cec033811f1975c0fda319450ca SEARCH MATCH TEXT ('content', '南方','topn=3');
 content
---------
(0 rows)

infiniflow=> select content from ragflow_df72e7dc032011f198f48f655838f485_ed351cec033811f1975c0fda319450ca SEARCH MATCH TEXT ('content', '南','topn=3');
 content
---------
(0 rows)

infiniflow=> select content from ragflow_df72e7dc032011f198f48f655838f485_ed351cec033811f1975c0fda319450ca SEARCH MATCH TEXT ('content', '北','topn=3');
 content
---------
(0 rows)

[dosubot]

This is a known issue with infinity as DOC_ENGINE: Chinese content is present in the database but cannot be retrieved due to how tokenization and analyzers are configured. The default analyzers (rag-coarse, rag-fine) do not segment Chinese text properly, so search queries in Chinese return no results even though the data exists. This is a tokenizer/analyzer mismatch problem—Chinese text requires segmentation that matches both at indexing and query time, but the current setup is optimized for English or whitespace-separated languages, not Chinese source source.

To address this, you should:

• Set LANG=zh_CN.UTF-8 in your environment before creating the knowledge base and restart the service. Changing this after KB creation does not update language handling retroactively source.
• Use a multilingual embedding model (like BAAI/BGE-m3, Qwen/text-embedding-v4, Cohere embed-multilingual, Voyage voyage-multilingual, or intfloat multilingual-e5). If you switch models, you must delete all existing chunks and re-ingest your documents source.
• Make sure the locale zh_CN.UTF-8 is actually installed inside your container or environment, not just set as an environment variable source.
• Avoid chunking modes that split every Chinese character (like "law" mode); use "General" chunking for better results source.
• Upgrade to the latest RAGFlow version if possible, as older versions had bugs blocking cross-language retrieval source.

Currently, true cross-language retrieval (e.g., querying in Chinese to retrieve English documents) is not fully supported; queries tend to match documents in the same language source.

If you need to support Chinese retrieval with infinity, you may need to customize the analyzer configuration for Chinese segmentation (if supported by Infinity), or consider using Elasticsearch with a Chinese analyzer like IK or smartcn for production scenarios.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[CAICAIIs]

Hi, I'm new to this repository. I'd like to try working on this issue.