feat(embedder): make embedder support openai compatible model like qwen (#169)

* feat(embedder): make embedder support openai compatible model like qwen

* refactor config py

---------

Co-authored-by: Xing Liu <[email protected]>

Author: XingLiu0923

README.es.md

@@ -239,6 +239,20 @@ La configuración de base_url del Cliente OpenAI está diseñada principalmente
 
 **Próximamente**: En futuras actualizaciones, DeepWiki soportará un modo donde los usuarios deberán proporcionar sus propias claves API en las solicitudes. Esto permitirá a los clientes empresariales con canales privados utilizar sus disposiciones API existentes sin compartir credenciales con el despliegue de DeepWiki.
 
+## 🧩 Uso de modelos de embedding compatibles con OpenAI (por ejemplo, Alibaba Qwen)
+
+Si deseas usar modelos de embedding compatibles con la API de OpenAI (como Alibaba Qwen), sigue estos pasos:
+
+1. Sustituye el contenido de `api/config/embedder.json` por el de `api/config/embedder_openai_compatible.json`.
+2. En el archivo `.env` de la raíz del proyecto, configura las variables de entorno necesarias, por ejemplo:
+   ```
+   OPENAI_API_KEY=tu_api_key
+   OPENAI_API_BASE_URL=tu_endpoint_compatible_openai
+   ```
+3. El programa sustituirá automáticamente los placeholders de embedder.json por los valores de tus variables de entorno.
+
+Así puedes cambiar fácilmente a cualquier servicio de embedding compatible con OpenAI sin modificar el código.
+
 ## 🤖 Funciones de Preguntas e Investigación Profunda
 
 ### Función de Preguntas
@@ -317,3 +331,4 @@ Este proyecto está licenciado bajo la Licencia MIT - consulta el archivo [LICEN
 ## ⭐ Historial de Estrellas
 
 [![Gráfico de Historial de Estrellas](https://api.star-history.com/svg?repos=AsyncFuncAI/deepwiki-open&type=Date)](https://star-history.com/#AsyncFuncAI/deepwiki-open&Date)
+

README.ja.md

@@ -444,3 +444,4 @@ _DeepWiki の動作を見る！_
 ## ⭐ スター履歴
 
 [![スター履歴チャート](https://api.star-history.com/svg?repos=AsyncFuncAI/deepwiki-open&type=Date)](https://star-history.com/#AsyncFuncAI/deepwiki-open&Date)
+

README.kr.md

@@ -427,3 +427,4 @@ DeepResearch를 사용하려면 질문 제출 전 Ask 인터페이스에서 "Dee
 ## ⭐ 스타 히스토리
 
 [![Star History Chart](https://api.star-history.com/svg?repos=AsyncFuncAI/deepwiki-open&type=Date)](https://star-history.com/#AsyncFuncAI/deepwiki-open&Date)
+

README.md

@@ -241,6 +241,20 @@ The OpenAI Client's base_url configuration is designed primarily for enterprise
 
 **Coming Soon**: In future updates, DeepWiki will support a mode where users need to provide their own API keys in requests. This will allow enterprise customers with private channels to use their existing API arrangements without sharing credentials with the DeepWiki deployment.
 
+## 🧩 Using OpenAI-Compatible Embedding Models (e.g., Alibaba Qwen)
+
+If you want to use embedding models compatible with the OpenAI API (such as Alibaba Qwen), follow these steps:
+
+1. Replace the contents of `api/config/embedder.json` with those from `api/config/embedder_openai_compatible.json`.
+2. In your project root `.env` file, set the relevant environment variables, for example:
+   ```
+   OPENAI_API_KEY=your_api_key
+   OPENAI_API_BASE_URL=your_openai_compatible_endpoint
+   ```
+3. The program will automatically substitute placeholders in embedder.json with the values from your environment variables.
+
+This allows you to seamlessly switch to any OpenAI-compatible embedding service without code changes.
+
 ## 🛠️ Advanced Setup
 
 ### Environment Variables

README.vi.md

@@ -420,3 +420,4 @@ Dự án này được cấp phép theo Giấy phép MIT - xem file [LICENSE](LI
 ## ⭐ Lịch sử
 
 [![Biểu đồ lịch sử](https://api.star-history.com/svg?repos=AsyncFuncAI/deepwiki-open&type=Date)](https://star-history.com/#AsyncFuncAI/deepwiki-open&Date)
+

README.zh.md

@@ -352,3 +352,18 @@ OpenAI 客户端的 base_url 配置主要为拥有私有 API 渠道的企业用
 - 支持与第三方 OpenAI API 兼容服务的集成
 
 **即将推出**：在未来的更新中，DeepWiki 将支持一种模式，用户需要在请求中提供自己的 API 密钥。这将允许拥有私有渠道的企业客户使用其现有的 API 安排，而不是与 DeepWiki 部署共享凭据。
+
+## 🧩 使用 OpenAI 兼容的 Embedding 模型（如阿里巴巴 Qwen）
+
+如果你希望使用 OpenAI 以外、但兼容 OpenAI 接口的 embedding 模型（如阿里巴巴 Qwen），请参考以下步骤：
+
+1. 用 `api/config/embedder_openai_compatible.json` 的内容替换 `api/config/embedder.json`。
+2. 在项目根目录的 `.env` 文件中，配置相应的环境变量，例如：
+   ```
+   OPENAI_API_KEY=你的_api_key
+   OPENAI_API_BASE_URL=你的_openai_兼容接口地址
+   ```
+3. 程序会自动用环境变量的值替换 embedder.json 里的占位符。
+
+这样即可无缝切换到 OpenAI 兼容的 embedding 服务，无需修改代码。
+

api/config.py

@@ -1,8 +1,9 @@
 import os
 import json
 import logging
+import re
 from pathlib import Path
-from typing import List
+from typing import List, Union, Dict, Any
 
 logger = logging.getLogger(__name__)
 
@@ -48,6 +49,36 @@
     "BedrockClient": BedrockClient
 }
 
+def replace_env_placeholders(config: Union[Dict[str, Any], List[Any], str, Any]) -> Union[Dict[str, Any], List[Any], str, Any]:
+    """
+    Recursively replace placeholders like "${ENV_VAR}" in string values
+    within a nested configuration structure (dicts, lists, strings)
+    with environment variable values. Logs a warning if a placeholder is not found.
+    """
+    pattern = re.compile(r"\$\{([A-Z0-9_]+)\}")
+
+    def replacer(match: re.Match[str]) -> str:
+        env_var_name = match.group(1)
+        original_placeholder = match.group(0)
+        env_var_value = os.environ.get(env_var_name)
+        if env_var_value is None:
+            logger.warning(
+                f"Environment variable placeholder '{original_placeholder}' was not found in the environment. "
+                f"The placeholder string will be used as is."
+            )
+            return original_placeholder
+        return env_var_value
+
+    if isinstance(config, dict):
+        return {k: replace_env_placeholders(v) for k, v in config.items()}
+    elif isinstance(config, list):
+        return [replace_env_placeholders(item) for item in config]
+    elif isinstance(config, str):
+        return pattern.sub(replacer, config)
+    else:
+        # Handles numbers, booleans, None, etc.
+        return config
+
 # Load JSON configuration file
 def load_json_config(filename):
     try:
@@ -65,7 +96,9 @@ def load_json_config(filename):
             return {}
 
         with open(config_path, 'r') as f:
-            return json.load(f)
+            config = json.load(f)
+            config = replace_env_placeholders(config)
+            return config
     except Exception as e:
         logger.error(f"Error loading configuration file {filename}: {str(e)}")
         return {}

api/config/embedder.json.bak

@@ -0,0 +1,25 @@
+{
+  "embedder": {
+    "client_class": "OpenAIClient",
+    "batch_size": 500,
+    "model_kwargs": {
+      "model": "text-embedding-3-small",
+      "dimensions": 256,
+      "encoding_format": "float"
+    }
+  },
+  "embedder_ollama": {
+    "client_class": "OllamaClient",
+    "model_kwargs": {
+      "model": "nomic-embed-text"
+    }
+  },
+  "retriever": {
+    "top_k": 20
+  },
+  "text_splitter": {
+    "split_by": "word",
+    "chunk_size": 350,
+    "chunk_overlap": 100
+  }
+}

api/config/embedder_openai_compatible.json.bak

@@ -0,0 +1,29 @@
+{
+  "embedder": {
+    "client_class": "OpenAIClient",
+    "initialize_kwargs": {
+      "api_key": "${OPENAI_API_KEY}",
+      "base_url": "${OPENAI_API_BASE_URL}"
+    },
+    "batch_size": 10,
+    "model_kwargs": {
+      "model": "text-embedding-v3",
+      "dimensions": 256,
+      "encoding_format": "float"
+    }
+  },
+  "embedder_ollama": {
+    "client_class": "OllamaClient",
+    "model_kwargs": {
+      "model": "nomic-embed-text"
+    }
+  },
+  "retriever": {
+    "top_k": 20
+  },
+  "text_splitter": {
+    "split_by": "word",
+    "chunk_size": 350,
+    "chunk_overlap": 100
+  }
+}

api/data_pipeline.py

@@ -15,6 +15,8 @@
 from api.ollama_patch import OllamaDocumentProcessor
 from urllib.parse import urlparse, urlunparse, quote
 
+from api.tools.embedder import get_embedder
+
 # Configure logging
 logger = logging.getLogger(__name__)
 
@@ -366,14 +368,7 @@ def prepare_data_pipeline(is_ollama_embedder: bool = None):
     splitter = TextSplitter(**configs["text_splitter"])
     embedder_config = get_embedder_config()
 
-    if not embedder_config:
-        raise ValueError("No embedder configuration found")
-
-    # Create embedder based on configuration
-    embedder = adal.Embedder(
-        model_client=embedder_config["model_client"](),
-        model_kwargs=embedder_config["model_kwargs"],
-    )
+    embedder = get_embedder(is_local_ollama=is_ollama_embedder)
 
     if is_ollama_embedder:
         # Use Ollama document processor for single-document processing