diff --git a/README.md b/README.md index 8867c394..7c773f21 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,15 @@

+

+ If you like Dingo, please give us a ⭐ on GitHub! +
+ + Click Star + +

+ + # Introduction Dingo is a data quality evaluation tool that helps you automatically detect data quality issues in your datasets. Dingo provides a variety of built-in rules and model evaluation methods, and also supports custom evaluation methods. Dingo supports commonly used text datasets and multimodal datasets, including pre-training datasets, fine-tuning datasets, and evaluation datasets. In addition, Dingo supports multiple usage methods, including local CLI and SDK, making it easy to integrate into various evaluation platforms, such as [OpenCompass](https://github.com/open-compass/opencompass). @@ -183,6 +192,7 @@ Our evaluation system includes: - **Classification Metrics**: Topic categorization and content classification - **Multimodality Assessment Metrics**: Image classification and relevance evaluation - **Rule-Based Quality Metrics**: Automated quality checks using heuristic rules for effectiveness and similarity detection +- **Factuality Assessment Metrics**: Two-stage factuality evaluation based on GPT-5 System Card - etc Most metrics are backed by academic sources to ensure objectivity and scientific rigor. @@ -217,6 +227,12 @@ For detailed guidance on using Dingo's hallucination detection capabilities, inc 📖 **[View Hallucination Detection Guide →](docs/hallucination_guide.md)** +### Factuality Assessment + +For comprehensive guidance on using Dingo's two-stage factuality evaluation system: + +📖 **[View Factuality Assessment Guide →](docs/factcheck_guide.md)** + # Rule Groups Dingo provides pre-configured rule groups for different types of datasets: diff --git a/README_ja.md b/README_ja.md index 96e2b179..29eab120 100644 --- a/README_ja.md +++ b/README_ja.md @@ -33,6 +33,14 @@ 👋 DiscordWeChatでご参加ください

+

+ このプロジェクトが役に立ったら、GitHubで⭐を付けてください! +
+ + Star をクリック + +

+ # はじめに @@ -178,6 +186,7 @@ Dingoはルールベースおよびプロンプトベースの評価メトリク - **分類メトリクス**: トピック分類とコンテンツ分類 - **マルチモーダル評価メトリクス**: 画像分類と関連性評価 - **ルールベース品質メトリクス**: ヒューリスティックルールによる効果性と類似性検出を用いた自動品質チェック +- **事実性評価メトリクス**: GPT-5 System Cardに基づく二段階事実性評価 - など 大部分のメトリクスは学術的なソースによって支持されており、客観性と科学的厳密性を保証しています。 @@ -212,6 +221,12 @@ HHEM-2.1-Openローカル推論とLLMベース評価を含む、Dingoの幻覚 📖 **[幻覚検出ガイドを見る →](docs/hallucination_guide.md)** +### 事実性評価 + +Dingoの二段階事実性評価システムの使用に関する詳細なガイダンス: + +📖 **[事実性評価ガイドを見る →](docs/factcheck_guide.md)** + # ルールグループ Dingoは異なるタイプのデータセット用に事前設定されたルールグループを提供します: diff --git a/README_zh-CN.md b/README_zh-CN.md index deaf4cca..484a24ab 100644 --- a/README_zh-CN.md +++ b/README_zh-CN.md @@ -29,6 +29,14 @@ 👋 加入我们 Discord微信

+

+ 如果觉得有帮助,欢迎在 GitHub 上点个 ⭐ 支持! +
+ + 点击 Star 支持 + +

+ @@ -179,6 +187,7 @@ Dingo通过基于规则和基于提示的评估指标提供全面的数据质量 - **分类指标**:主题分类和内容分类 - **多模态评估指标**:图像分类和相关性评估 - **基于规则的质量指标**:使用启发式规则进行效果性和相似性检测的自动化质量检查 +- **事实性评估指标**:基于 GPT-5 System Card 的两阶段事实性评估 - 等等 大部分指标都由学术来源支持,以确保客观性和科学严谨性。 @@ -213,6 +222,12 @@ input_data = { 📖 **[查看幻觉检测指南 →](docs/hallucination_guide.md)** +### 事实性评估 + +有关使用Dingo两阶段事实性评估系统的详细指导: + +📖 **[查看事实性评估指南 →](docs/factcheck_guide.md)** + # 规则组 Dingo为不同类型的数据集提供预配置的规则组: diff --git a/app_gradio/app.py b/app_gradio/app.py index 680bde77..6520bb17 100644 --- a/app_gradio/app.py +++ b/app_gradio/app.py @@ -48,20 +48,26 @@ def dingo_demo( try: input_data = { - "dataset": dataset_source, - "data_format": data_format, "input_path": final_input_path, "output_path": "" if dataset_source == 'hugging_face' else os.path.dirname(final_input_path), - "save_data": True, - "save_raw": True, - - "max_workers": max_workers, - "batch_size": batch_size, - - "column_content": column_content, - "custom_config": { + "dataset": { + "source": dataset_source, + "format": data_format, + "field": { + "content": column_content + } + }, + "executor": { "rule_list": rule_list, "prompt_list": prompt_list, + "result_save": { + "bad": True, + "raw": True + }, + "max_workers": max_workers, + "batch_size": batch_size, + }, + "evaluator": { "llm_config": { scene_list: { "model": model, @@ -72,11 +78,11 @@ def dingo_demo( } } if column_id: - input_data['column_id'] = column_id + input_data['dataset']['field']['id'] = column_id if column_prompt: - input_data['column_prompt'] = column_prompt + input_data['dataset']['field']['prompt'] = column_prompt if column_image: - input_data['column_image'] = column_image + input_data['dataset']['field']['image'] = column_image # print(input_data) # exit(0) diff --git a/app_gradio/header.html b/app_gradio/header.html index 78f11d45..b0800b05 100644 --- a/app_gradio/header.html +++ b/app_gradio/header.html @@ -67,7 +67,7 @@ color: #fafafa; opacity: 0.8; "> - Dingo: A Comprehensive Data Quality Evaluation Tool.
+ Dingo: A Comprehensive AI Data Quality Evaluation Tool.