Documents API wrapper vs LLM-native toolkits; includes Slack API wrapper toolkit docs #435
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…ned, not how they are used or called
|
|
||
| [LLM-native tools](/home/use-tools/types-of-tools#llm-native-tools) are designed from scratch to provide the best performance for LLMs in terms of speed, reliability, accuracy, and cost-effectiveness. | ||
|
|
||
| #### API wrapper tools |
There was a problem hiding this comment.
I think it can be beneficial to call these something different. I iterated a bit with LLMs, and I like these candidates:
- Semantic API Adapters tools (maybe too long)
- API Proxy tools (my favorite)
- Intent-API tools
There was a problem hiding this comment.
out of those three I like API Proxy as well
|
|
||
| #### API wrapper tools | ||
|
|
||
| [API wrapper tools](/home/use-tools/types-of-tools#api-wrapper-tools) are designed to mirror the original API design of the third-party service. They are not optimized for LLM usage and are not subject to evaluation suites. We recommend thoroughly evaluating each API wrapper tool with your Agents or chatbots before using it in production. |
There was a problem hiding this comment.
| [API wrapper tools](/home/use-tools/types-of-tools#api-wrapper-tools) are designed to mirror the original API design of the third-party service. They are not optimized for LLM usage and are not subject to evaluation suites. We recommend thoroughly evaluating each API wrapper tool with your Agents or chatbots before using it in production. | |
| [API wrapper tools](/home/use-tools/types-of-tools#api-wrapper-tools) are designed to mirror the original API design of the third-party service. They are not fully optimized for LLM usage and are not subject to evaluation suites. We apply basic principles of MXE to increase the probability of LLMs calling them effectively. Nevertheless, we recommend thoroughly evaluating each API wrapper tool with your Agents or chatbots before using it in production. |
There was a problem hiding this comment.
I thought about mentioning MXE, but then we need to explain that term as well
do you think we should have a separate page about MXE or a topic in this page explaining it? would also need a glossary entry
it would be interesting to start establishing a position of thought leadership in that space...
There was a problem hiding this comment.
Yes I think a dedicated page for MXE will be useful later on, but it's not necessary at the moment. Until we need it, we can point to your blog post or one of the interviews
|
|
||
| Since they require careful design and evaluation, LLM-native tools take time and effort to build. We understand that your Agent or chatbot project might need capabilities not yet covered by our LLM-native toolkits. For this reason, we also offer low-level API wrapper toolkits. | ||
|
|
||
| ## API wrapper tools |
There was a problem hiding this comment.
Update the new name here, (and link it) from the glossary
| Arcade offers two types of tools: | ||
|
|
||
| - API wrapper tools | ||
| - LLM-native tools |
| @@ -47,6 +47,14 @@ const availableTools: Tool[] = [ | |||
| category: "social", | |||
| type: "arcade", | |||
| }, | |||
| { | |||
| name: "Slack API Wrapper", | |||
Co-authored-by: Mateo Torres <[email protected]>
| 3. Open a direct message between the user and John, and get the channel ID | ||
| 4. Send the message to the channel | ||
|
|
||
| Even the most powerful LLMs usually perform poorly when they need to reason such complex workflows on the fly, not to mention the increased cost and risk of hallucinations. As a result, AI Agents and chatbots that rely on HTTP APIs often end up being unreliable. |
There was a problem hiding this comment.
Moving away from generalities like "usually perform poorly", "rarely matches the data model", etc., to specific metrics will help drive the point home.
I think it would be really cool to do an eval benchmark for one API wrapper toolkit vs one LLM-native toolkit (for the same upstream provider) to show how much better LLM-native is. We could show a bar chart and start saying things like "performs 76% more poorly", etc.
There was a problem hiding this comment.
That would be awesome indeed
I'll think of a simple benchmark we can start with
| @@ -47,6 +47,14 @@ const availableTools: Tool[] = [ | |||
| category: "social", | |||
| type: "arcade", | |||
| }, | |||
| { | |||
| name: "Slack API Wrapper", | |||
There was a problem hiding this comment.
It almost feels like API wrappers should be in an entirely different list separate from LLM-native toolkits.
There was a problem hiding this comment.
Perhaps we should add a filter in the catalog view?
The benefit of keeping them together is facilitating discovery, I think. Otherwise users need to browse two places and they may miss one after finding the other.
There was a problem hiding this comment.
At the very least, we should visually represent wrappers with a color or badge to distinguish them as "different" in this list. I also like the filter idea!
The main files to review are:
ApiWrapperInfocomponent to display a notice about api wrapper: /src/components/ApiWrapperInfo.tsx