Skip to content

Conversation

@MichalKalita
Copy link
Contributor

Closes https://github.com/apify/ai-team/issues/30

It's a simple test to run Agent with a prompt and check how the tools were used.
The target is to be sure the tool description is good and LLM doesn't make mistakes with tool selection and arguments.

@github-actions github-actions bot added t-ai Issues owned by the AI team. tested Temporary label used only programatically for some analytics. labels Jul 2, 2025
@MichalKalita
Copy link
Contributor Author

This MCP server is not compatible with the Mastra MCP client. We allow adding tools (Actors) at runtime inside LLM requests, while Mastra only allows adding tools between requests. This may be a problem for more clients.

We have two options:

  1. Implement Add generic call-actor tool #155
  2. Use on MCP client other than Mastra ( but it's itself a problem, when popular MCP clients don't work, it's itself an evaluation fail)

@jirispilka @MQ37

@MQ37
Copy link
Contributor

MQ37 commented Jul 3, 2025

This MCP server is not compatible with the Mastra MCP client. We allow adding tools (Actors) at runtime inside LLM requests, while Mastra only allows adding tools between requests. This may be a problem for more clients.

We have two options:

1. Implement [Add generic call-actor tool #155](https://github.com/apify/actors-mcp-server/issues/155)

2. Use on MCP client other than Mastra ( but it's itself a problem, when popular MCP clients don't work, it's itself an evaluation fail)

@jirispilka @MQ37

I think this conclusion makes sense, I think we should implement the generic call-actor tool 👍

@jirispilka
Copy link
Collaborator

@MichalKalita is this PR still relevant since we have generic call-actor tool

@MichalKalita
Copy link
Contributor Author

@jirispilka I'm closing this PR. We want a tool that allows us to conduct A/B tests in complex scenarios, collect all metrics, and decide which way is better.

@MichalKalita MichalKalita deleted the feature/evaluation branch July 16, 2025 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-ai Issues owned by the AI team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants