fix: update dataset

Author: jirispilka

evals/README.md

@@ -143,6 +143,13 @@ To get the best performance out of Claude when using tools, follow these guideli
 - **Prioritize descriptions over examples.**
   While you can include examples of how to use a tool in its description or accompanying prompt, this is less important than having a clear and comprehensive explanation of the tool’s purpose and parameters.
   Only add examples **after** you’ve fully developed the description.
+
+## Optimize metadata for OpenAI models
+
+- Name – pair the domain with the action (calendar.create_event).
+- Description – start with “Use this when…” and call out disallowed cases (“Do not use for reminders”).
+- Parameter docs – describe each argument, include examples, and use enums for constrained values.
+- Read-only hint – annotate readOnlyHint: true on tools that never mutate state so ChatGPT can streamline confirmation.
 ---
 
 ## How to analyze and improve a specific tool
@@ -212,6 +219,43 @@ Always make improvements **manually**, based on your understanding of the proble
 LLMs are very likely to worsen the issue instead of fixing it.
 
 
-# References:
+# Tool definition patterns
+
+Based on analysis of [Cursor Agent Tools v1.0](https://raw.githubusercontent.com/x1xhlol/system-prompts-and-models-of-ai-tools/refs/heads/main/Cursor%20Prompts/Agent%20Tools%20v1.0.json), [Lovable Agent Tools](https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools/blob/main/Lovable/Agent%20Tools.json), and [Claude Code Tools](https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools/blob/main/Claude%20Code/claude-code-tools.json):
+
+## Tool description vs parameter description
+
+**Tool description** should contain:
+- What the tool does (core functionality)
+- When to use it (usage context)
+- Key limitations (what it doesn't do)
+- High-level behavior (how it works conceptually)
+
+**Parameter description** should contain:
+- Parameter-specific details (what each parameter does)
+- Input constraints (validation rules, formats)
+- Usage examples (specific examples for that parameter)
+- Parameter-specific guidance (how to use that specific parameter)
+
+## Key patterns
+
+1. **Concise but comprehensive** - Avoid overly verbose descriptions
+2. **Semantic clarity** - Use language that matches user intent
+3. **Clear separation** - Tool purpose vs parameter-specific guidance
+4. **Operational constraints** - State limitations and boundaries
+5. **Contextual guidance** - Include usage instructions where relevant
+
+## References
 
 - [Example of a good tool description](https://docs.claude.com/en/docs/agents-and-tools/tool-use/implement-tool-use#example-of-a-good-tool-description)
+- [Cursor Agent Tools v1.0](https://raw.githubusercontent.com/x1xhlol/system-prompts-and-models-of-ai-tools/refs/heads/main/Cursor%20Prompts/Agent%20Tools%20v1.0.json)
+- [Lovable Agent Tools](https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools/blob/main/Lovable/Agent%20Tools.json)
+- [Claude Code Tools](https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools/blob/main/Claude%20Code/claude-code-tools.json)
+- [OpenAI optimize metadata](https://developers.openai.com/apps-sdk/guides/optimize-metadata)
+
+NOTES:
+
+// System prompt - instructions mainly cursor (very similar instructions in copilot)
+// https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools/blob/main/Cursor%20Prompts/Agent%20Prompt%20v1.2.txt
+// https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools/blob/main/VSCode%20Agent/Prompt.txt
+

evals/test-cases.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.1",
+  "version": "1.3",
   "testCases": [
     {
       "id": "fetch-actor-details-1",
@@ -66,15 +66,15 @@
       "id": "search-actors-1",
       "category": "search-actors",
       "query": "How to scrape Instagram posts",
-      "expectedTools": ["search-actors"],
-      "reference": "It must call the 'search-actors' tool with the query: 'Instagram posts' or similar"
+      "expectedTools": [],
+      "reference": "Either it should explain how to scrape Instagram posts or call 'search-actors' tool with the query: 'Instagram posts' or similar"
     },
     {
       "id": "search-actors-2",
       "category": "search-actors",
       "query": "What are the best Instagram scrapers?",
       "expectedTools": ["search-actors"],
-      "reference": "It must call the 'search-actors' tool with the query: 'Instagram scraper' or similar."
+      "reference": "It must call the 'search-actors' tool with the query: `Instagram`, 'Instagram scraper', or similar."
     },
     {
       "id": "search-actors-3",
@@ -196,9 +196,9 @@
     {
       "id": "search-vs-rag-1",
       "category": "search-actors",
-      "query": "Find posts about AI on Instagram",
+      "query": "Find posts about the Rock on Instagram",
       "expectedTools": ["search-actors"],
-      "reference": "It must call the 'search-actors' tool with the query: 'Instagram' or 'Instagram posts' or similar. It must not use extended queries such as 'Instagram posts AI' or any more detailed variations."
+      "reference": "It must call the 'search-actors' tool with the query: 'Instagram' or 'Instagram posts' or similar. It must not use extended queries such as 'Instagram posts the Rock' or any more detailed variations."
     },
     {
       "id": "search-vs-rag-2",
@@ -234,9 +234,8 @@
     {
       "id": "search-vs-rag-7",
       "category": "search-actors",
-      "query": "Fetch flight details for New York to London",
-      "expectedTools": ["search-actors"],
-      "reference": "It must call the 'search-actors' tool with the query: 'flight data' or 'flight booking' or similar"
+      "query": "Find one way flights from New York to London tomorrow",
+      "expectedTools": ["search-actors"]
     },
     {
       "id": "search-vs-rag-8",
@@ -391,8 +390,8 @@
     },
     {
       "id": "misleading-query-1",
-      "category": "search-actors",
-      "query": "What's the weather like today?",
+      "category": "apify-slash-rag-web-browser",
+      "query": "What's the weather like today in San Francisco?",
       "expectedTools": ["apify-slash-rag-web-browser"]
     },
     {
@@ -410,7 +409,7 @@
     {
       "id": "ambiguous-query-1",
       "category": "search-actors",
-      "query": "Instagram posts",
+      "query": "Get instagram posts",
       "expectedTools": ["search-actors"],
       "reference": "It must call the 'search-actors' tool with the query: 'Instagram posts' or similar"
     },