The App can't understand the Chinese documents

[tzuhsin0329]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

1. build a VM(Win10) in Azure and use it to deploy the APP.
2. simply run azd login、azd init -t azure-search-openai-demo
3. put the Chinese documents in ./data folder.
4. run azd up and ./script/prepdocs.ps1 to split the documents and upload to blob and insert imformation to search index.
5. open the endpoint and start to use the app.
6. issue：can't answer the question based on the Chinese document.

Any log messages given by the failure

No.

Expected/desired behavior

Only can read English documents and answer. Can't read Chinese documents and answer the questions.

OS and Version?

Windows 10

Versions

0eed611

Mention any other details that might be useful

Try to change the analyzer in cognitive search service to Chinese, but the app still can't read Chinese documents, the search service can. I have tried the cognitive service on Azure portal.

---

Thanks! We'll be in touch soon.

[sissimonster]

I didn't do any changes but it seems mine can read both. May I know what is the respond you got when you ask the Chinese question?

[XunLi-Nick]

You can try modifying the prompt of app\backend\approvals\chateadretrieveread.py by changing "If the question is not in English, translate the question to English before generating the search query." to "Please search in the language of the original input of the question, never try to translate it into English."

[danwalsh-ses]

I added the following function to chatreadretrieveread.py
def is_chinese_query(q): for char in q: category = unicodedata.category(char) if category == 'Lo' or 'CJK' in category: return True return False
and then in Step 2 in the same file I changed the self.search_client.search parameters to check for chinese within the query_language and query_speller parameters. This should cover Korean and Japanese since it's checking CJK but I still need to test but it is working for Chinese documents now. However, it returns answers in English so I am updating the prompt to tell it to return in the original input language. Need to confirm this.
r = self.search_client.search(q, filter=filter, query_type=QueryType.SEMANTIC, query_language="zh-cn" if is_chinese else "en-us", query_speller="lexicon" if not is_chinese else None, semantic_configuration_name="default", top=top, query_caption="extractive|highlight-false" if use_semantic_captions else None)

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this issue will be closed.

[pamelafox]

I'm closing this issue as the application has undergone significant improvements since 2023 that should address Chinese document support:

1. CJK Text Processing: The document ingestion pipeline (prepdocslib/textsplitter.py) now includes dedicated support for CJK sentence endings (。, ！, ？, etc.) and word breaks, ensuring proper chunking of Chinese text during document parsing and indexing.

2. Improved Embeddings: The application now supports modern embedding models that handle multilingual text more effectively.

If you're still experiencing issues with Chinese documents in the current version, please open a new issue with:

• The version/commit you're using
• Sample error messages
• Details about your Azure OpenAI model configuration
• Whether the issue is with document ingestion or query/response

If you'd like to contribute Chinese UI translations (the interface currently supports 10 languages but not Chinese), we welcome PRs adding Chinese locale files following the pattern in app/frontend/src/locales/.