Merge pull request #171 from e06084/main

sync dev to main

Author: e06084

README.md

@@ -36,6 +36,15 @@
 </p>
 
 
+<p align="center">
+  If you like Dingo, please give us a ⭐ on GitHub!
+  <br/>
+  <a href="https://github.com/DataEval/dingo/stargazers" target="_blank">
+    <img src="docs/assets/clickstar_2.gif" alt="Click Star" width="480">
+  </a>
+</p>
+
+
 # Introduction
 
 Dingo is a data quality evaluation tool that helps you automatically detect data quality issues in your datasets. Dingo provides a variety of built-in rules and model evaluation methods, and also supports custom evaluation methods. Dingo supports commonly used text datasets and multimodal datasets, including pre-training datasets, fine-tuning datasets, and evaluation datasets. In addition, Dingo supports multiple usage methods, including local CLI and SDK, making it easy to integrate into various evaluation platforms, such as [OpenCompass](https://github.com/open-compass/opencompass).
@@ -183,6 +192,7 @@ Our evaluation system includes:
 - **Classification Metrics**: Topic categorization and content classification
 - **Multimodality Assessment Metrics**: Image classification and relevance evaluation
 - **Rule-Based Quality Metrics**: Automated quality checks using heuristic rules for effectiveness and similarity detection
+- **Factuality Assessment Metrics**: Two-stage factuality evaluation based on GPT-5 System Card
 - etc
 
 Most metrics are backed by academic sources to ensure objectivity and scientific rigor.
@@ -217,6 +227,12 @@ For detailed guidance on using Dingo's hallucination detection capabilities, inc
 
 📖 **[View Hallucination Detection Guide →](docs/hallucination_guide.md)**
 
+### Factuality Assessment
+
+For comprehensive guidance on using Dingo's two-stage factuality evaluation system:
+
+📖 **[View Factuality Assessment Guide →](docs/factcheck_guide.md)**
+
 # Rule Groups
 
 Dingo provides pre-configured rule groups for different types of datasets:

README_ja.md

@@ -33,6 +33,14 @@
     👋 <a href="https://discord.gg/Jhgb2eKWh8" target="_blank">Discord</a>と<a href="./docs/assets/wechat.jpg" target="_blank">WeChat</a>でご参加ください
 </p>
 
+<p align="center">
+  このプロジェクトが役に立ったら、GitHubで⭐を付けてください！
+  <br/>
+  <a href="https://github.com/DataEval/dingo/stargazers" target="_blank">
+    <img src="docs/assets/clickstar_2.gif" alt="Star をクリック" width="480">
+  </a>
+</p>
+
 
 # はじめに
 
@@ -178,6 +186,7 @@ Dingoはルールベースおよびプロンプトベースの評価メトリク
 - **分類メトリクス**: トピック分類とコンテンツ分類
 - **マルチモーダル評価メトリクス**: 画像分類と関連性評価
 - **ルールベース品質メトリクス**: ヒューリスティックルールによる効果性と類似性検出を用いた自動品質チェック
+- **事実性評価メトリクス**: GPT-5 System Cardに基づく二段階事実性評価
 - など
 
 大部分のメトリクスは学術的なソースによって支持されており、客観性と科学的厳密性を保証しています。
@@ -212,6 +221,12 @@ HHEM-2.1-Openローカル推論とLLMベース評価を含む、Dingoの幻覚
 
 📖 **[幻覚検出ガイドを見る →](docs/hallucination_guide.md)**
 
+### 事実性評価
+
+Dingoの二段階事実性評価システムの使用に関する詳細なガイダンス：
+
+📖 **[事実性評価ガイドを見る →](docs/factcheck_guide.md)**
+
 # ルールグループ
 
 Dingoは異なるタイプのデータセット用に事前設定されたルールグループを提供します：

README_zh-CN.md

@@ -29,6 +29,14 @@
     👋 加入我们 <a href="https://discord.gg/Jhgb2eKWh8" target="_blank">Discord</a> 和 <a href="./docs/assets/wechat.jpg" target="_blank">微信</a>
 </p>
 
+<p align="center">
+  如果觉得有帮助，欢迎在 GitHub 上点个 ⭐ 支持！
+  <br/>
+  <a href="https://github.com/DataEval/dingo/stargazers" target="_blank">
+    <img src="docs/assets/clickstar_2.gif" alt="点击 Star 支持" width="480">
+  </a>
+</p>
+
 </div>
 
 
@@ -179,6 +187,7 @@ Dingo通过基于规则和基于提示的评估指标提供全面的数据质量
 - **分类指标**：主题分类和内容分类
 - **多模态评估指标**：图像分类和相关性评估
 - **基于规则的质量指标**：使用启发式规则进行效果性和相似性检测的自动化质量检查
+- **事实性评估指标**：基于 GPT-5 System Card 的两阶段事实性评估
 - 等等
 
 大部分指标都由学术来源支持，以确保客观性和科学严谨性。
@@ -213,6 +222,12 @@ input_data = {
 
 📖 **[查看幻觉检测指南 →](docs/hallucination_guide.md)**
 
+### 事实性评估
+
+有关使用Dingo两阶段事实性评估系统的详细指导：
+
+📖 **[查看事实性评估指南 →](docs/factcheck_guide.md)**
+
 # 规则组
 
 Dingo为不同类型的数据集提供预配置的规则组：

app_gradio/app.py

@@ -48,20 +48,26 @@ def dingo_demo(
 
     try:
         input_data = {
-            "dataset": dataset_source,
-            "data_format": data_format,
             "input_path": final_input_path,
             "output_path": "" if dataset_source == 'hugging_face' else os.path.dirname(final_input_path),
-            "save_data": True,
-            "save_raw": True,
-
-            "max_workers": max_workers,
-            "batch_size": batch_size,
-
-            "column_content": column_content,
-            "custom_config": {
+            "dataset": {
+                "source": dataset_source,
+                "format": data_format,
+                "field": {
+                    "content": column_content
+                }
+            },
+            "executor": {
                 "rule_list": rule_list,
                 "prompt_list": prompt_list,
+                "result_save": {
+                    "bad": True,
+                    "raw": True
+                },
+                "max_workers": max_workers,
+                "batch_size": batch_size,
+            },
+            "evaluator": {
                 "llm_config": {
                     scene_list: {
                         "model": model,
@@ -72,11 +78,11 @@ def dingo_demo(
             }
         }
         if column_id:
-            input_data['column_id'] = column_id
+            input_data['dataset']['field']['id'] = column_id
         if column_prompt:
-            input_data['column_prompt'] = column_prompt
+            input_data['dataset']['field']['prompt'] = column_prompt
         if column_image:
-            input_data['column_image'] = column_image
+            input_data['dataset']['field']['image'] = column_image
 
         # print(input_data)
         # exit(0)

app_gradio/header.html

@@ -67,7 +67,7 @@
           color: #fafafa;
           opacity: 0.8;
         ">
-        Dingo: A Comprehensive Data Quality Evaluation Tool.<br>
+        Dingo: A Comprehensive AI Data Quality Evaluation Tool.<br>
       </p>
       <style>
         .link-block {

dingo/exec/local.py

@@ -173,6 +173,11 @@ def evaluate(self):
         log.debug("[Summary]: " + str(self.summary))
 
     def evaluate_single_data(self, group_type, group, data: Data):
+        # Ensure dynamic configs are applied in child processes as well
+        try:
+            Model.apply_config(self.input_args)
+        except Exception as e:
+            raise RuntimeError(f"Failed to apply config in child process: {e}")
         result_info = ResultInfo(
             data_id=data.data_id, prompt=data.prompt, content=data.content
         )