Skip to content

Commit 931ea82

Browse files
authored
Merge pull request #74 from DataEval/dev
Realese v1.6
2 parents 877a380 + 61d7bac commit 931ea82

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+1528
-374
lines changed

.github/workflows/IntegrationTest.yml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@ name: Python application
55

66
on:
77
push:
8-
branches: [ "main", "dev" ]
8+
branches: [ "*" ]
99
pull_request:
10-
branches: [ "main", "dev" ]
10+
branches: [ "*" ]
1111
workflow_dispatch:
1212

1313

@@ -37,6 +37,7 @@ jobs:
3737
- name: Integration Test(local plaintext)
3838
run: |
3939
python -m dingo.run.cli --input_path test/data/test_local_plaintext.txt --dataset local -e default --data_format plaintext
40+
python -m dingo.run.cli --input_path test/data/test_local_plaintext.txt --dataset local -e default --data_format plaintext --save_data
4041
- name: Integration Test(local json)
4142
run: |
4243
python -m dingo.run.cli --input_path test/data/test_local_json.json --dataset local -e default --data_format json --column_content prediction

README.md

Lines changed: 63 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,19 @@
1717

1818
</div>
1919

20-
[English](README.md) | [简体中文](README_zh-CN.md)
20+
21+
<div align="center">
22+
23+
[English](README.md) · [简体中文](README_zh-CN.md)
24+
25+
</div>
26+
2127

2228
<div align="center">
2329
<a href="https://discord.gg/Jhgb2eKWh8" style="text-decoration:none;">
2430
<img src="https://user-images.githubusercontent.com/25839884/218347213-c080267f-cbb6-443e-8532-8e1ed9a58ea9.png" width="3%" alt="Discord" /></a>
25-
<img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" />
2631
<a href="https://huggingface.co/spaces/DataEval/dingo" style="text-decoration:none;">
2732
<img src="https://huggingface.co/datasets/huggingface/brand-assets/resolve/main/hf-logo.png" width="3%" alt="Hugging Face" /></a>
28-
<img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" />
2933
</div>
3034

3135

@@ -51,7 +55,43 @@ pip install dingo-python
5155

5256
## Example Use Cases
5357

54-
### 1. Evaluate Local Text File (Plaintext)
58+
### 1. Using Evaluate Core
59+
60+
```python
61+
from dingo.config.config import DynamicLLMConfig
62+
from dingo.io.input.MetaData import MetaData
63+
from dingo.model.llm.llm_text_quality_model_base import LLMTextQualityModelBase
64+
from dingo.model.rule.rule_common import RuleEnterAndSpace
65+
66+
67+
def llm():
68+
data = MetaData(
69+
data_id='123',
70+
prompt="hello, introduce the world",
71+
content="Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty."
72+
)
73+
74+
LLMTextQualityModelBase.dynamic_config = DynamicLLMConfig(
75+
key='',
76+
api_url='',
77+
# model='',
78+
)
79+
res = LLMTextQualityModelBase.eval(data)
80+
print(res)
81+
82+
83+
def rule():
84+
data = MetaData(
85+
data_id='123',
86+
prompt="hello, introduce the world",
87+
content="Hello! The world is a vast and diverse place, full of wonders, cultures, and incredible natural beauty."
88+
)
89+
90+
res = RuleEnterAndSpace().eval(data)
91+
print(res)
92+
```
93+
94+
### 2. Evaluate Local Text File (Plaintext)
5595

5696
```python
5797
from dingo.io import InputArgs
@@ -72,7 +112,7 @@ result = executor.execute()
72112
print(result)
73113
```
74114

75-
### 2. Evaluate Hugging Face Dataset
115+
### 3. Evaluate Hugging Face Dataset
76116

77117
```python
78118
from dingo.io import InputArgs
@@ -92,7 +132,7 @@ result = executor.execute()
92132
print(result)
93133
```
94134

95-
### 3. Evaluate JSON/JSONL Format
135+
### 4. Evaluate JSON/JSONL Format
96136

97137
```python
98138
from dingo.io import InputArgs
@@ -114,7 +154,7 @@ result = executor.execute()
114154
print(result)
115155
```
116156

117-
### 4. Using LLM for Evaluation
157+
### 5. Using LLM for Evaluation
118158

119159
```python
120160
from dingo.io import InputArgs
@@ -229,6 +269,7 @@ Dingo provides several LLM-based assessment methods defined by prompts in the `d
229269
|-------------|--------|-------------|
230270
| `TEXT_QUALITY_KAOTI` | Exam question quality | Specialized assessment for evaluating the quality of exam questions, focusing on formula rendering, table formatting, paragraph structure, and answer formatting |
231271
| `Html_Abstract` | HTML extraction quality | Compares different methods of extracting Markdown from HTML, evaluating completeness, formatting accuracy, and semantic coherence |
272+
| `DATAMAN_ASSESSMENT` | Data Quality & Domain | Evaluates pre-training data quality using the DataMan methodology (14 standards, 15 domains). Assigns a score (0/1), domain type, quality status, and reason. |
232273

233274
### Classification Prompts
234275

@@ -420,6 +461,21 @@ Example summary:
420461
}
421462
```
422463

464+
465+
# MCP Server (Experimental)
466+
467+
Dingo includes an experimental Model Context Protocol (MCP) server. For details on running the server and integrating it with clients like Cursor, please see the dedicated documentation:
468+
469+
[**Dingo MCP Server Documentation (README_mcp.md)**](README_mcp.md)
470+
471+
472+
# Research & Publications
473+
474+
- **"Comprehensive Data Quality Assessment for Multilingual WebData"** : [WanJuanSiLu: A High-Quality Open-Source Webtext
475+
Dataset for Low-Resource Languages](https://arxiv.org/pdf/2501.14506)
476+
- **"Pre-training data quality using the DataMan methodology"** : [DataMan: Data Manager for Pre-training Large Language Models](https://openreview.net/pdf?id=eNbA8Fqir4)
477+
478+
423479
# Future Plans
424480

425481
- [ ] Richer graphic and text evaluation indicators

README_mcp.md

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# Dingo MCP Server
2+
3+
## Overview
4+
5+
The `mcp_server.py` script provides an experimental Model Context Protocol (MCP) server for Dingo, powered by [FastMCP](https://github.com/modelcontextprotocol/fastmcp). This allows MCP clients, such as Cursor, to interact with Dingo's data evaluation capabilities programmatically.
6+
7+
## Features
8+
9+
* Exposes Dingo's evaluation logic via MCP.
10+
* Provides two primary tools:
11+
* `run_dingo_evaluation`: Executes rule-based or LLM-based evaluations on specified data.
12+
* `list_dingo_components`: Lists available rule groups and registered LLM models within Dingo.
13+
* Enables interaction through MCP clients like Cursor.
14+
15+
## Installation
16+
17+
1. **Prerequisites**: Ensure you have Git and a Python environment (e.g., 3.8+) set up.
18+
2. **Clone the Repository**: Clone this repository to your local machine.
19+
```bash
20+
git clone https://github.com/DataEval/dingo.git
21+
cd dingo
22+
```
23+
3. **Install Dependencies**: Install the required dependencies, including FastMCP and other Dingo requirements. It's recommended to use the `requirements.txt` file.
24+
```bash
25+
pip install -r requirements.txt
26+
# Alternatively, at minimum: pip install fastmcp
27+
```
28+
4. **Ensure Dingo is Importable**: Make sure your Python environment can find the `dingo` package within the cloned repository when you run the server script.
29+
30+
## Running the Server
31+
32+
Navigate to the directory containing `mcp_server.py` and run it using Python:
33+
34+
```bash
35+
python mcp_server.py
36+
```
37+
38+
By default, the server starts using the Server-Sent Events (SSE) transport protocol. You can customize its behavior using arguments within the script's `mcp.run()` call:
39+
40+
```python
41+
# Example customization in mcp_server.py
42+
mcp.run(
43+
transport="sse", # Communication protocol (sse is default)
44+
host="127.0.0.1", # Network interface to bind to (default: 0.0.0.0)
45+
port=8888, # Port to listen on (default: 8000)
46+
log_level="debug" # Logging verbosity (default: info)
47+
)
48+
```
49+
50+
**Important**: Note the `host` and `port` the server is running on, as you will need these to configure your MCP client.
51+
52+
## Integration with Cursor
53+
54+
### Configuration
55+
56+
To connect Cursor to your running Dingo MCP server, you need to edit Cursor's MCP configuration file (`mcp.json`). This file is typically located in Cursor's user configuration directory (e.g., `~/.cursor/` or `%USERPROFILE%\.cursor\`).
57+
58+
Add or modify the entry for your Dingo server within the `mcpServers` object. Use the `url` property to specify the address of your running server.
59+
60+
**Example `mcp.json` entry:**
61+
62+
```json
63+
{
64+
"mcpServers": {
65+
// ... other servers ...
66+
"dingo_evaluator": {
67+
"url": "http://127.0.0.1:8888/sse" // <-- MUST match host, port, and transport of your running server
68+
}
69+
// ...
70+
}
71+
}
72+
```
73+
74+
* Ensure the `url` exactly matches the `host`, `port`, and `transport` (currently only `sse` is supported for the URL scheme) your `mcp_server.py` is configured to use. If you didn't customize `mcp.run`, the default URL is likely `http://127.0.0.1:8000/sse` or `http://0.0.0.0:8000/sse`.
75+
* Restart Cursor after saving changes to `mcp.json`.
76+
77+
### Usage in Cursor
78+
79+
Once configured, you can invoke the Dingo tools within Cursor:
80+
81+
* **List Components**: "Use the dingo_evaluator tool to list available Dingo components."
82+
* **Run Evaluation**: "Use the dingo_evaluator tool to run a rule evaluation..." or "Use the dingo_evaluator tool to run an LLM evaluation..."
83+
84+
Cursor will prompt you for the necessary arguments.
85+
86+
## Tool Reference
87+
88+
### `list_dingo_components()`
89+
90+
Lists available Dingo rule groups and registered LLM model identifiers.
91+
92+
* **Arguments**: None
93+
* **Returns**: `Dict[str, List[str]]` - A dictionary containing `rule_groups` and `llm_models`.
94+
95+
**Example Cursor Usage**:
96+
> Use the dingo_evaluator tool to list dingo components.
97+
98+
### `run_dingo_evaluation(...)`
99+
100+
Runs a Dingo evaluation (rule-based or LLM-based).
101+
102+
* **Arguments**:
103+
* `input_path` (str): Path to the input file or directory (relative to the project root or absolute).
104+
* `evaluation_type` (Literal["rule", "llm"]): Type of evaluation.
105+
* `eval_group_name` (str): Rule group name for `rule` type (default: `""` which uses 'default'). Only 'default', 'sft', 'pretrain' are validated by the server logic. Ignored for `llm` type.
106+
* `output_dir` (Optional[str]): Directory to save outputs. Defaults to a `dingo_output_*` subdirectory within the parent directory of `input_path`.
107+
* `task_name` (Optional[str]): Name for the task (used in output path generation). Defaults to `mcp_eval_<uuid>`.
108+
* `save_data` (bool): Whether to save detailed JSONL output (default: True).
109+
* `save_correct` (bool): Whether to save correct data (default: True).
110+
* `kwargs` (dict): Dictionary for additional `dingo.io.InputArgs`. Common uses:
111+
* `dataset` (str): Dataset type (e.g., 'local', 'hugging_face'). Defaults to 'local' if `input_path` is given.
112+
* `data_format` (str): Input data format (e.g., 'json', 'jsonl', 'plaintext'). Inferred from `input_path` extension if possible.
113+
* `column_content` (str): **Required** for formats like JSON/JSONL - specifies the key containing the text to evaluate.
114+
* `column_id`, `column_prompt`, `column_image`: Other column mappings.
115+
* `custom_config` (str | dict): Path to a JSON config file, a JSON string, or a dictionary for LLM evaluation or custom rule settings. API keys for LLMs **must** be provided here.
116+
* `max_workers`, `batch_size`: Dingo execution parameters (default to 1 in MCP for stability).
117+
* **Returns**: `str` - The absolute path to the primary output file (e.g., `summary.json`).
118+
119+
**Example Cursor Usage (Rule-based):**
120+
121+
> Use the Dingo Evaluator tool to run the default rule evaluation on `test/data/test_local_jsonl.jsonl`. Make sure to use the 'content' column.
122+
123+
*(Cursor should propose a tool call like below)*
124+
```xml
125+
<use_mcp_tool>
126+
<server_name>dingo_evaluator</server_name>
127+
<tool_name>run_dingo_evaluation</tool_name>
128+
<arguments>
129+
{
130+
"input_path": "test/data/test_local_jsonl.jsonl",
131+
"evaluation_type": "rule",
132+
"eval_group_name": "default",
133+
"kwargs": {
134+
"column_content": "content"
135+
// data_format="jsonl" and dataset="local" will be inferred
136+
}
137+
}
138+
</arguments>
139+
</use_mcp_tool>
140+
```
141+
142+
**Example Cursor Usage (LLM-based):**
143+
144+
> Use the Dingo Evaluator tool to perform an LLM evaluation on `test/data/test_local_jsonl.jsonl`. Use the 'content' column. Configure it using the file `examples/mcp/config_self_deployed_llm.json`.
145+
146+
*(Cursor should propose a tool call like below. Note `eval_group_name` can be omitted or set when using `custom_config` for LLM evals)*
147+
```xml
148+
<use_mcp_tool>
149+
<server_name>dingo_evaluator</server_name>
150+
<tool_name>run_dingo_evaluation</tool_name>
151+
<arguments>
152+
{
153+
"input_path": "test/data/test_local_jsonl.jsonl",
154+
"evaluation_type": "llm",
155+
"kwargs": {
156+
"column_content": "content",
157+
"custom_config": "examples/mcp/config_self_deployed_llm.json"
158+
// data_format="jsonl" and dataset="local" will be inferred
159+
}
160+
}
161+
</arguments>
162+
</use_mcp_tool>
163+
```
164+
165+
Refer to `examples/mcp/config_api_llm.json` (for API-based LLMs) and `examples/mcp/config_self_deployed_llm.json` (for self-hosted LLMs) for the structure of the `custom_config` file, including where to place API keys or URLs.

0 commit comments

Comments
 (0)