|
| 1 | +# Dingo MCP Server |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +The `mcp_server.py` script provides an experimental Model Context Protocol (MCP) server for Dingo, powered by [FastMCP](https://github.com/modelcontextprotocol/fastmcp). This allows MCP clients, such as Cursor, to interact with Dingo's data evaluation capabilities programmatically. |
| 6 | + |
| 7 | +## Features |
| 8 | + |
| 9 | +* Exposes Dingo's evaluation logic via MCP. |
| 10 | +* Provides two primary tools: |
| 11 | + * `run_dingo_evaluation`: Executes rule-based or LLM-based evaluations on specified data. |
| 12 | + * `list_dingo_components`: Lists available rule groups and registered LLM models within Dingo. |
| 13 | +* Enables interaction through MCP clients like Cursor. |
| 14 | + |
| 15 | +## Installation |
| 16 | + |
| 17 | +1. **Prerequisites**: Ensure you have Git and a Python environment (e.g., 3.8+) set up. |
| 18 | +2. **Clone the Repository**: Clone this repository to your local machine. |
| 19 | + ```bash |
| 20 | + git clone https://github.com/DataEval/dingo.git |
| 21 | + cd dingo |
| 22 | + ``` |
| 23 | +3. **Install Dependencies**: Install the required dependencies, including FastMCP and other Dingo requirements. It's recommended to use the `requirements.txt` file. |
| 24 | + ```bash |
| 25 | + pip install -r requirements.txt |
| 26 | + # Alternatively, at minimum: pip install fastmcp |
| 27 | + ``` |
| 28 | +4. **Ensure Dingo is Importable**: Make sure your Python environment can find the `dingo` package within the cloned repository when you run the server script. |
| 29 | +
|
| 30 | +## Running the Server |
| 31 | +
|
| 32 | +Navigate to the directory containing `mcp_server.py` and run it using Python: |
| 33 | +
|
| 34 | +```bash |
| 35 | +python mcp_server.py |
| 36 | +``` |
| 37 | +
|
| 38 | +By default, the server starts using the Server-Sent Events (SSE) transport protocol. You can customize its behavior using arguments within the script's `mcp.run()` call: |
| 39 | +
|
| 40 | +```python |
| 41 | +# Example customization in mcp_server.py |
| 42 | +mcp.run( |
| 43 | + transport="sse", # Communication protocol (sse is default) |
| 44 | + host="127.0.0.1", # Network interface to bind to (default: 0.0.0.0) |
| 45 | + port=8888, # Port to listen on (default: 8000) |
| 46 | + log_level="debug" # Logging verbosity (default: info) |
| 47 | +) |
| 48 | +``` |
| 49 | + |
| 50 | +**Important**: Note the `host` and `port` the server is running on, as you will need these to configure your MCP client. |
| 51 | + |
| 52 | +## Integration with Cursor |
| 53 | + |
| 54 | +### Configuration |
| 55 | + |
| 56 | +To connect Cursor to your running Dingo MCP server, you need to edit Cursor's MCP configuration file (`mcp.json`). This file is typically located in Cursor's user configuration directory (e.g., `~/.cursor/` or `%USERPROFILE%\.cursor\`). |
| 57 | + |
| 58 | +Add or modify the entry for your Dingo server within the `mcpServers` object. Use the `url` property to specify the address of your running server. |
| 59 | + |
| 60 | +**Example `mcp.json` entry:** |
| 61 | + |
| 62 | +```json |
| 63 | +{ |
| 64 | + "mcpServers": { |
| 65 | + // ... other servers ... |
| 66 | + "dingo_evaluator": { |
| 67 | + "url": "http://127.0.0.1:8888/sse" // <-- MUST match host, port, and transport of your running server |
| 68 | + } |
| 69 | + // ... |
| 70 | + } |
| 71 | +} |
| 72 | +``` |
| 73 | + |
| 74 | +* Ensure the `url` exactly matches the `host`, `port`, and `transport` (currently only `sse` is supported for the URL scheme) your `mcp_server.py` is configured to use. If you didn't customize `mcp.run`, the default URL is likely `http://127.0.0.1:8000/sse` or `http://0.0.0.0:8000/sse`. |
| 75 | +* Restart Cursor after saving changes to `mcp.json`. |
| 76 | + |
| 77 | +### Usage in Cursor |
| 78 | + |
| 79 | +Once configured, you can invoke the Dingo tools within Cursor: |
| 80 | + |
| 81 | +* **List Components**: "Use the dingo_evaluator tool to list available Dingo components." |
| 82 | +* **Run Evaluation**: "Use the dingo_evaluator tool to run a rule evaluation..." or "Use the dingo_evaluator tool to run an LLM evaluation..." |
| 83 | + |
| 84 | +Cursor will prompt you for the necessary arguments. |
| 85 | + |
| 86 | +## Tool Reference |
| 87 | + |
| 88 | +### `list_dingo_components()` |
| 89 | + |
| 90 | +Lists available Dingo rule groups and registered LLM model identifiers. |
| 91 | + |
| 92 | +* **Arguments**: None |
| 93 | +* **Returns**: `Dict[str, List[str]]` - A dictionary containing `rule_groups` and `llm_models`. |
| 94 | + |
| 95 | +**Example Cursor Usage**: |
| 96 | +> Use the dingo_evaluator tool to list dingo components. |
| 97 | +
|
| 98 | +### `run_dingo_evaluation(...)` |
| 99 | + |
| 100 | +Runs a Dingo evaluation (rule-based or LLM-based). |
| 101 | + |
| 102 | +* **Arguments**: |
| 103 | + * `input_path` (str): Path to the input file or directory (relative to the project root or absolute). |
| 104 | + * `evaluation_type` (Literal["rule", "llm"]): Type of evaluation. |
| 105 | + * `eval_group_name` (str): Rule group name for `rule` type (default: `""` which uses 'default'). Only 'default', 'sft', 'pretrain' are validated by the server logic. Ignored for `llm` type. |
| 106 | + * `output_dir` (Optional[str]): Directory to save outputs. Defaults to a `dingo_output_*` subdirectory within the parent directory of `input_path`. |
| 107 | + * `task_name` (Optional[str]): Name for the task (used in output path generation). Defaults to `mcp_eval_<uuid>`. |
| 108 | + * `save_data` (bool): Whether to save detailed JSONL output (default: True). |
| 109 | + * `save_correct` (bool): Whether to save correct data (default: True). |
| 110 | + * `kwargs` (dict): Dictionary for additional `dingo.io.InputArgs`. Common uses: |
| 111 | + * `dataset` (str): Dataset type (e.g., 'local', 'hugging_face'). Defaults to 'local' if `input_path` is given. |
| 112 | + * `data_format` (str): Input data format (e.g., 'json', 'jsonl', 'plaintext'). Inferred from `input_path` extension if possible. |
| 113 | + * `column_content` (str): **Required** for formats like JSON/JSONL - specifies the key containing the text to evaluate. |
| 114 | + * `column_id`, `column_prompt`, `column_image`: Other column mappings. |
| 115 | + * `custom_config` (str | dict): Path to a JSON config file, a JSON string, or a dictionary for LLM evaluation or custom rule settings. API keys for LLMs **must** be provided here. |
| 116 | + * `max_workers`, `batch_size`: Dingo execution parameters (default to 1 in MCP for stability). |
| 117 | +* **Returns**: `str` - The absolute path to the primary output file (e.g., `summary.json`). |
| 118 | + |
| 119 | +**Example Cursor Usage (Rule-based):** |
| 120 | + |
| 121 | +> Use the Dingo Evaluator tool to run the default rule evaluation on `test/data/test_local_jsonl.jsonl`. Make sure to use the 'content' column. |
| 122 | +
|
| 123 | +*(Cursor should propose a tool call like below)* |
| 124 | +```xml |
| 125 | +<use_mcp_tool> |
| 126 | +<server_name>dingo_evaluator</server_name> |
| 127 | +<tool_name>run_dingo_evaluation</tool_name> |
| 128 | +<arguments> |
| 129 | +{ |
| 130 | + "input_path": "test/data/test_local_jsonl.jsonl", |
| 131 | + "evaluation_type": "rule", |
| 132 | + "eval_group_name": "default", |
| 133 | + "kwargs": { |
| 134 | + "column_content": "content" |
| 135 | + // data_format="jsonl" and dataset="local" will be inferred |
| 136 | + } |
| 137 | +} |
| 138 | +</arguments> |
| 139 | +</use_mcp_tool> |
| 140 | +``` |
| 141 | + |
| 142 | +**Example Cursor Usage (LLM-based):** |
| 143 | + |
| 144 | +> Use the Dingo Evaluator tool to perform an LLM evaluation on `test/data/test_local_jsonl.jsonl`. Use the 'content' column. Configure it using the file `examples/mcp/config_self_deployed_llm.json`. |
| 145 | +
|
| 146 | +*(Cursor should propose a tool call like below. Note `eval_group_name` can be omitted or set when using `custom_config` for LLM evals)* |
| 147 | +```xml |
| 148 | +<use_mcp_tool> |
| 149 | +<server_name>dingo_evaluator</server_name> |
| 150 | +<tool_name>run_dingo_evaluation</tool_name> |
| 151 | +<arguments> |
| 152 | +{ |
| 153 | + "input_path": "test/data/test_local_jsonl.jsonl", |
| 154 | + "evaluation_type": "llm", |
| 155 | + "kwargs": { |
| 156 | + "column_content": "content", |
| 157 | + "custom_config": "examples/mcp/config_self_deployed_llm.json" |
| 158 | + // data_format="jsonl" and dataset="local" will be inferred |
| 159 | + } |
| 160 | +} |
| 161 | +</arguments> |
| 162 | +</use_mcp_tool> |
| 163 | +``` |
| 164 | + |
| 165 | +Refer to `examples/mcp/config_api_llm.json` (for API-based LLMs) and `examples/mcp/config_self_deployed_llm.json` (for self-hosted LLMs) for the structure of the `custom_config` file, including where to place API keys or URLs. |
0 commit comments