feat: 优化webkit抽取prompt

README.md

@@ -1,7 +1,18 @@
-<div align="center" xmlns="http://www.w3.org/1999/html">
+<!-- SEO Meta Information and Structured Data -->
+<div itemscope itemtype="https://schema.org/SoftwareApplication" align="center" xmlns="http://www.w3.org/1999/html">
+  <meta itemprop="name" content="Dingo: A Comprehensive AI Data Quality Evaluation Tool">
+  <meta itemprop="description" content="Comprehensive AI-powered data quality assessment platform for machine learning datasets, LLM training data validation, hallucination detection, and RAG system evaluation">
+  <meta itemprop="applicationCategory" content="Data Quality Software">
+  <meta itemprop="operatingSystem" content="Cross-platform">
+  <meta itemprop="programmingLanguage" content="Python">
+  <meta itemprop="url" content="https://github.com/MigoXLab/dingo">
+  <meta itemprop="downloadUrl" content="https://pypi.org/project/dingo-python/">
+  <meta itemprop="softwareVersion" content="latest">
+  <meta itemprop="license" content="Apache-2.0">
+
 <!-- logo -->
 <p align="center">
-  <img src="docs/assets/dingo-logo.png" width="300px" style="vertical-align:middle;">
+  <img src="docs/assets/dingo-logo.png" width="300px" style="vertical-align:middle;" alt="Dingo AI Data Quality Evaluation Tool Logo">
 </p>
 
 <!-- badges -->
@@ -15,8 +26,7 @@
   <a href="https://github.com/DataEval/dingo/issues"><img src="https://img.shields.io/github/issues/DataEval/dingo" alt="GitHub issues"></a>
   <a href="https://mseep.ai/app/dataeval-dingo"><img src="https://mseep.net/pr/dataeval-dingo-badge.png" alt="MseeP.ai Security Assessment Badge" height="20"></a>
   <a href="https://deepwiki.com/MigoXLab/dingo"><img src="https://deepwiki.com/badge.svg" alt="Ask DeepWiki"></a>
-
-[![Trust Score](https://archestra.ai/mcp-catalog/api/badge/quality/DataEval/dingo)](https://archestra.ai/mcp-catalog/dataeval__dingo)
+  <a href="https://archestra.ai/mcp-catalog/dataeval__dingo"><img src="https://archestra.ai/mcp-catalog/api/badge/quality/DataEval/dingo" alt="Trust Score"></a>
 </p>
 
 </div>
@@ -45,7 +55,7 @@
 </p>
 
 
-# Introduction
+# Introduction of Dingo
 
 Dingo is a data quality evaluation tool that helps you automatically detect data quality issues in your datasets. Dingo provides a variety of built-in rules and model evaluation methods, and also supports custom evaluation methods. Dingo supports commonly used text datasets and multimodal datasets, including pre-training datasets, fine-tuning datasets, and evaluation datasets. In addition, Dingo supports multiple usage methods, including local CLI and SDK, making it easy to integrate into various evaluation platforms, such as [OpenCompass](https://github.com/open-compass/opencompass).
 
@@ -61,7 +71,7 @@ Dingo is a data quality evaluation tool that helps you automatically detect data
 pip install dingo-python
 ```
 
-## Example Use Cases
+## Example Use Cases of Dingo
 
 ### 1. Evaluate LLM chat data
 

README_ja.md

@@ -1,7 +1,18 @@
-<div align="center" xmlns="http://www.w3.org/1999/html">
+<!-- SEO メタ情報と構造化データ -->
+<div itemscope itemtype="https://schema.org/SoftwareApplication" align="center" xmlns="http://www.w3.org/1999/html">
+  <meta itemprop="name" content="Dingo: AI データ品質評価ツール">
+  <meta itemprop="description" content="機械学習データセット、LLM学習データ検証、幻覚検出、RAGシステム評価のための包括的なAI駆動データ品質評価プラットフォーム">
+  <meta itemprop="applicationCategory" content="データ品質ソフトウェア">
+  <meta itemprop="operatingSystem" content="クロスプラットフォーム">
+  <meta itemprop="programmingLanguage" content="Python">
+  <meta itemprop="url" content="https://github.com/MigoXLab/dingo">
+  <meta itemprop="downloadUrl" content="https://pypi.org/project/dingo-python/">
+  <meta itemprop="softwareVersion" content="latest">
+  <meta itemprop="license" content="Apache-2.0">
+
 <!-- logo -->
 <p align="center">
-  <img src="docs/assets/dingo-logo.png" width="300px" style="vertical-align:middle;">
+  <img src="docs/assets/dingo-logo.png" width="300px" style="vertical-align:middle;" alt="Dingo AI データ品質評価ツール ロゴ">
 </p>
 
 <!-- badges -->

README_zh-CN.md

@@ -1,7 +1,18 @@
-<div align="center" xmlns="http://www.w3.org/1999/html">
+<!-- SEO 元信息和结构化数据 -->
+<div itemscope itemtype="https://schema.org/SoftwareApplication" align="center" xmlns="http://www.w3.org/1999/html">
+  <meta itemprop="name" content="Dingo: AI 数据质量评估工具">
+  <meta itemprop="description" content="全面的AI驱动数据质量评估平台，专为机器学习数据集、LLM训练数据验证、幻觉检测和RAG系统评估而设计">
+  <meta itemprop="applicationCategory" content="数据质量软件">
+  <meta itemprop="operatingSystem" content="跨平台">
+  <meta itemprop="programmingLanguage" content="Python">
+  <meta itemprop="url" content="https://github.com/MigoXLab/dingo">
+  <meta itemprop="downloadUrl" content="https://pypi.org/project/dingo-python/">
+  <meta itemprop="softwareVersion" content="latest">
+  <meta itemprop="license" content="Apache-2.0">
+
 <!-- logo -->
 <p align="center">
-  <img src="docs/assets/dingo-logo.png" width="300px" style="vertical-align:middle;">
+  <img src="docs/assets/dingo-logo.png" width="300px" style="vertical-align:middle;" alt="Dingo AI 数据质量评估工具 Logo">
 </p>
 
 <!-- badges -->
@@ -40,7 +51,7 @@
 </div>
 
 
-# 介绍
+# Dingo 介绍
 
 Dingo是一款数据质量评估工具，帮助你自动化检测数据集中的数据质量问题。Dingo提供了多种内置的规则和模型评估方法，同时也支持自定义评估方法。Dingo支持常用的文本数据集和多模态数据集，包括预训练数据集、微调数据集和评测数据集。此外，Dingo支持多种使用方式，包括本地CLI和SDK，便于集成到各种评测平台，如[OpenCompass](https://github.com/open-compass/opencompass)等。
 
@@ -57,7 +68,7 @@ Dingo是一款数据质量评估工具，帮助你自动化检测数据集中的
 pip install dingo-python
 ```
 
-## 2. 使用示例
+## 2. Dingo 使用示例
 
 ### 2.1 评估LLM对话数据
 

app_gradio/app.py

@@ -438,4 +438,4 @@ def get_data_column_mapping():
         )
 
     # 启动界面
-    demo.launch()
+    demo.launch(server_port=7861, share=True)

dingo/data/datasource/local.py

@@ -30,20 +30,51 @@ def load_local_file(path: str, by_line: bool = True) -> Generator[str, None, Non
     Returns:
         str: The contents of the file.
     """
+    import gzip
+
     if not os.path.exists(path):
         raise RuntimeError(f'"{path}" is not a valid path')
     f_list = []
     if os.path.exists(path) and os.path.isfile(path):
         f_list = [path]
     elif os.path.exists(path) and os.path.isdir(path):
         find_all_files(path, f_list)
+
     for f in f_list:
-        with open(f, "r", encoding="utf-8") as _f:
-            if by_line:
-                for line in _f.readlines():
-                    yield line
-            else:
-                yield _f.read()
+        # Check if file is gzipped
+        if f.endswith('.gz'):
+            try:
+                with gzip.open(f, 'rt', encoding='utf-8') as _f:
+                    if by_line:
+                        for line in _f.readlines():
+                            yield line
+                    else:
+                        yield _f.read()
+            except Exception as gz_error:
+                raise RuntimeError(
+                    f'Failed to read gzipped file "{f}": {str(gz_error)}. '
+                    f'Please ensure the file is a valid gzip-compressed text file.'
+                )
+        else:
+            # For regular files, try UTF-8 encoding
+            try:
+                with open(f, "r", encoding="utf-8") as _f:
+                    if by_line:
+                        for line in _f.readlines():
+                            yield line
+                    else:
+                        yield _f.read()
+            except UnicodeDecodeError as decode_error:
+                raise RuntimeError(
+                    f'Failed to read file "{f}": Unsupported file format or encoding. '
+                    f'Dingo only supports UTF-8 text files (.jsonl, .json, .txt) and .gz compressed text files. '
+                    f'Original error: {str(decode_error)}'
+                )
+            except Exception as e:
+                raise RuntimeError(
+                    f'Unexpected error reading file "{f}": {str(e)}. '
+                    f'Please check if the file exists and is readable.'
+                )
 
 
 @DataSource.register()

dingo/exec/local.py

@@ -238,12 +238,30 @@ def evaluate_rule(self, group: List[BaseRule], d: Data) -> ResultInfo:
             # analyze result
             if tmp.error_status:
                 result_info.error_status = True
-                bad_type_list.append(tmp.type)
-                bad_name_list.append(tmp.type + "-" + tmp.name)
+                if isinstance(tmp.type, str) and isinstance(tmp.name, str):
+                    bad_type_list.append(tmp.type)
+                    bad_name_list.append(tmp.type + "-" + tmp.name)
+                elif isinstance(tmp.type, List) and isinstance(tmp.name, List):
+                    if len(tmp.type) != len(tmp.name):
+                        raise Exception(f'ModelRes.type is not the same length to ModelRes.name.\n type: {tmp.type} \n name: {tmp.name}')
+                    for i in range(len(tmp.type)):
+                        bad_type_list.append(tmp.type[i])
+                        bad_name_list.append(tmp.type[i] + "-" + tmp.name[i])
+                else:
+                    raise Exception('ModelRes.type and ModelRes.name are not str or List at the same time.')
                 bad_reason_list.extend(tmp.reason)
             else:
-                good_type_list.append(tmp.type)
-                good_name_list.append(tmp.type + "-" + tmp.name)
+                if isinstance(tmp.type, str) and isinstance(tmp.name, str):
+                    good_type_list.append(tmp.type)
+                    good_name_list.append(tmp.type + "-" + tmp.name)
+                elif isinstance(tmp.type, List) and isinstance(tmp.name, List):
+                    if len(tmp.type) != len(tmp.name):
+                        raise Exception(f'ModelRes.type is not the same length to ModelRes.name.\n type: {tmp.type} \n name: {tmp.name}')
+                    for i in range(len(tmp.type)):
+                        good_type_list.append(tmp.type[i])
+                        good_name_list.append(tmp.type[i] + "-" + tmp.name[i])
+                else:
+                    raise Exception('ModelRes.type and ModelRes.name are not str or List at the same time.')
                 good_reason_list.extend(tmp.reason)
         if result_info.error_status:
             result_info.type_list = list(set(bad_type_list))
@@ -271,12 +289,32 @@ def evaluate_prompt(self, group: List[BasePrompt], d: Data) -> ResultInfo:
             # analyze result
             if tmp.error_status:
                 result_info.error_status = True
-                bad_type_list.append(tmp.type)
-                bad_name_list.append(tmp.type + "-" + tmp.name)
+                if isinstance(tmp.type, str) and isinstance(tmp.name, str):
+                    bad_type_list.append(tmp.type)
+                    bad_name_list.append(tmp.type + "-" + tmp.name)
+                elif isinstance(tmp.type, List) and isinstance(tmp.name, List):
+                    if len(tmp.type) != len(tmp.name):
+                        raise Exception(
+                            f'ModelRes.type is not the same length to ModelRes.name.\n type: {tmp.type} \n name: {tmp.name}')
+                    for i in range(len(tmp.type)):
+                        bad_type_list.append(tmp.type[i])
+                        bad_name_list.append(tmp.type[i] + "-" + tmp.name[i])
+                else:
+                    raise Exception('ModelRes.type and ModelRes.name are not str or List at the same time.')
                 bad_reason_list.extend(tmp.reason)
             else:
-                good_type_list.append(tmp.type)
-                good_name_list.append(tmp.type + "-" + tmp.name)
+                if isinstance(tmp.type, str) and isinstance(tmp.name, str):
+                    good_type_list.append(tmp.type)
+                    good_name_list.append(tmp.type + "-" + tmp.name)
+                elif isinstance(tmp.type, List) and isinstance(tmp.name, List):
+                    if len(tmp.type) != len(tmp.name):
+                        raise Exception(
+                            f'ModelRes.type is not the same length to ModelRes.name.\n type: {tmp.type} \n name: {tmp.name}')
+                    for i in range(len(tmp.type)):
+                        good_type_list.append(tmp.type[i])
+                        good_name_list.append(tmp.type[i] + "-" + tmp.name[i])
+                else:
+                    raise Exception('ModelRes.type and ModelRes.name are not str or List at the same time.')
                 good_reason_list.extend(tmp.reason)
         if result_info.error_status:
             result_info.type_list = list(set(bad_type_list))

dingo/model/llm/llm_factcheck_public.py

@@ -99,6 +99,9 @@ def eval(cls, input_data: Data) -> ModelRes:
 
         except Exception as e:
             return ModelRes(
+                error_status=True,
+                type="QUALITY_BAD_FACTUALITY",
+                name="FACTUALITY_CHECK_ERROR",
                 score=0.0,
                 threshold=cls.threshold,
                 reason=[f"Evaluation failed: {str(e)}"],
@@ -210,41 +213,23 @@ def _parse_check_results(cls, text: str) -> List[FactCheckResult]:
 
             results = []
             for item in data:
-                evidence_list = [
-                    Evidence(**e) for e in item["supporting_evidence"]
-                ]
+                # 处理 evidence，确保所有必需字段都存在
+                evidence_list = []
+                for e in item.get("supporting_evidence", []):
+                    # 确保所有必需字段都存在，提供默认值
+                    evidence = Evidence(
+                        url=e.get("url", ""),
+                        snippet=e.get("snippet", ""),  # 提供默认值避免缺失
+                        summary=e.get("summary", "")
+                    )
+                    evidence_list.append(evidence)
+
                 results.append(FactCheckResult(
-                    claim=item["claim"],
-                    answer=item["answer"],
-                    reasoning=item["reasoning"],
+                    claim=item.get("claim", ""),
+                    answer=item.get("answer", "unsure"),  # 默认为 unsure
+                    reasoning=item.get("reasoning", ""),
                     supporting_evidence=evidence_list
                 ))
             return results
         except Exception as e:
             raise ValueError(f"Invalid results format: {str(e)}")
-
-    @classmethod
-    def send_messages(cls, messages: List) -> str:
-        """重写发送消息方法，避免使用 models.list()"""
-        if not cls.dynamic_config.model:
-            raise ValueError("model name must be specified")
-
-        params = cls.dynamic_config.parameters or {}
-        cls.validate_config(params)
-
-        completions = cls.client.chat.completions.create(
-            model=cls.dynamic_config.model,
-            messages=messages,
-            temperature=params.get("temperature", 0.3),
-            top_p=params.get("top_p", 1),
-            max_tokens=params.get("max_tokens", 4000),
-            presence_penalty=params.get("presence_penalty", 0),
-            frequency_penalty=params.get("frequency_penalty", 0),
-        )
-
-        if completions.choices[0].finish_reason == "length":
-            raise ExceedMaxTokens(
-                f"Exceed max tokens: {params.get('max_tokens', 4000)}"
-            )
-
-        return str(completions.choices[0].message.content)

dingo/model/llm/llm_hallucination.py

@@ -58,7 +58,7 @@ def build_messages(cls, input_data: Data) -> List:
         # Format contexts for display
         contexts_str = json.dumps(contexts, ensure_ascii=False, indent=2)
 
-        prompt_content = cls.prompt.content % (question, response, contexts_str)
+        prompt_content = cls.prompt.content.format(question, response, contexts_str)
 
         messages = [{"role": "user", "content": prompt_content}]
         return messages

dingo/model/llm/llm_html_abtract.py

@@ -6,15 +6,15 @@
 from dingo.model import Model
 from dingo.model.llm.base_openai import BaseOpenAI
 from dingo.model.modelres import ModelRes
-from dingo.model.prompt.prompt_html_abstract import PromptHtmlAbstract
+from dingo.model.prompt.prompt_html_extract_compare import PromptHtmlExtractCompare
 from dingo.model.response.response_class import ResponseScoreTypeNameReason
 from dingo.utils import log
 from dingo.utils.exception import ConvertJsonError
 
 
-@Model.llm_register("LLMHtmlAbstract")
-class LLMHtmlAbstract(BaseOpenAI):
-    prompt = PromptHtmlAbstract
+@Model.llm_register("LLMHtmlExtractCompare")
+class LLMHtmlExtractCompare(BaseOpenAI):
+    prompt = PromptHtmlExtractCompare
 
     @classmethod
     def build_messages(cls, input_data: Data) -> List:
@@ -23,8 +23,8 @@ def build_messages(cls, input_data: Data) -> List:
                 "role": "user",
                 "content": cls.prompt.content.format(
                     input_data.content,
-                    input_data.raw_data["markdown_ours"],
-                    input_data.raw_data["markdown_m10"],
+                    input_data.raw_data["magic_md"],
+                    input_data.raw_data["content"],
                 ),
             }
         ]

dingo/model/llm/llm_text_3h.py

@@ -41,12 +41,12 @@ def process_response(cls, response: str) -> ModelRes:
 
         # error_status
         if response_model.score == 1:
-            result.reason = [response_model.reason]
+            result.reason = [response_model.reason] if response_model.reason else ["Response meets quality criteria"]
             result.name = cls.prompt.__name__[8:].upper()
         else:
             result.error_status = True
             result.type = "QUALITY_BAD"
-            result.reason = [response_model.reason]
+            result.reason = [response_model.reason] if response_model.reason else ["Response fails quality criteria"]
             result.name = "NOT_" + cls.prompt.__name__[8:].upper()
 
         return result