Skip to content

fix: improve MCP tool descriptions and evaluations #2099

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
atinux merged 7 commits into from
Nov 27, 2025
Merged

fix: improve MCP tool descriptions and evaluations #2099

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
812bdff
improve MCP tool descriptions and add evals
onmax Nov 18, 2025
b6a6241
Merge branch 'main' into improve-mcp-tool-descriptions
HugoRCD Nov 20, 2025
480576f
Merge remote-tracking branch 'origin/main' into improve-mcp-tool-desc…
HugoRCD Nov 21, 2025
0073df0
chore(deps): resolve merge conflict with main
onmax Nov 25, 2025
679f277
chore(deps): merge main
onmax Nov 25, 2025
61eb372
docs: add instructions for running evals on the MCP server
HugoRCD Nov 26, 2025
0d1a522
improve stucture and leverage AI Gateway
atinux Nov 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,7 @@ NUXT_OAUTH_GITHUB_CLIENT_SECRET=
# Session encryption password (generate a secure random string)
# You can use: openssl rand -base64 32
NUXT_SESSION_PASSWORD=

# MCP Evaluation (Optional - for running evalite tests)
OPENAI_API_KEY=
MCP_URL=http://localhost:3000/mcp
28 changes: 28 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,34 @@ Build the application for production:
pnpm generate
```

### Running Evals for the MCP Server

To run the evals for the MCP server, follow these steps:

1. **Ensure your development server is running**
Start the local Nuxt development server:
```bash
pnpm dev
```

2. **Create an AI Gateway API key**
Go to https://vercel.com/ai-gateway and create an API key.
Add the following variable to your `.env` file (replace `sk-...` with your actual key):
```
AI_GATEWAY_API_KEY=<you-api-key>
```

3. **Run the evals**
You can execute the evals from the command line:
```bash
pnpm eval
```

Or launch the interactive UI to run them via a web interface:
```bash
pnpm eval:ui
```

## License

[MIT License](./LICENSE)
6 changes: 6 additions & 0 deletions evalite.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import 'dotenv/config'
import { defineConfig } from 'evalite/config'

export default defineConfig({
cache: true
})
6 changes: 6 additions & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
"lint": "eslint . --cache",
"typecheck": "nuxt typecheck",
"test": "pnpm lint && pnpm typecheck",
"eval": "evalite",
"eval:ui": "evalite serve",
"db:generate": "nuxt hub database generate",
"db:migrate": "nuxt hub database migrate"
},
Expand Down Expand Up @@ -54,15 +56,19 @@
"valibot": "1.1.0"
},
"devDependencies": {
"@ai-sdk/mcp": "^0.0.8",
"@ai-sdk/openai": "3.0.0-beta.60",
"@iconify-json/vscode-icons": "^1.2.36",
"@nuxt/eslint": "^1.10.0",
"@nuxt/test-utils": "^3.20.1",
"@nuxtjs/turnstile": "^1.1.1",
"@testing-library/vue": "^8.1.0",
"@types/youtube": "^0.1.2",
"ai": "6.0.0-beta.99",
"capture-website": "^5.1.0",
"drizzle-kit": "^0.31.7",
"eslint": "^9.39.1",
"evalite": "1.0.0-beta.13",
"nuxt-content-twoslash": "0.1.2",
"shiki": "^3.15.0",
"twoslash": "^0.3.4",
Expand Down
Loading