-
Notifications
You must be signed in to change notification settings - Fork 4
Documents API wrapper vs LLM-native toolkits; includes Slack API wrapper toolkit docs #435
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
c1566d4
c09e6a5
384844e
a1adc09
c13c47a
1ae0e6a
9108633
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -30,6 +30,16 @@ Tools are commonly referred to by a qualified name that includes their toolkit. | |||||
|
||||||
*Learn more about [tools](/home/build-tools/create-a-toolkit).* | ||||||
|
||||||
#### LLM-native tools | ||||||
|
||||||
[LLM-native tools](/home/use-tools/types-of-tools#llm-native-tools) are designed from scratch to provide the best performance for LLMs in terms of speed, reliability, accuracy, and cost-effectiveness. | ||||||
|
||||||
#### API wrapper tools | ||||||
|
||||||
[API wrapper tools](/home/use-tools/types-of-tools#api-wrapper-tools) are designed to mirror the original API design of the third-party service. They are not optimized for LLM usage and are not subject to evaluation suites. We recommend thoroughly evaluating each API wrapper tool with your Agents or chatbots before using it in production. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I thought about mentioning MXE, but then we need to explain that term as well do you think we should have a separate page about MXE or a topic in this page explaining it? would also need a glossary entry it would be interesting to start establishing a position of thought leadership in that space... There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes I think a dedicated page for MXE will be useful later on, but it's not necessary at the moment. Until we need it, we can point to your blog post or one of the interviews
byrro marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
Understand why [LLMs usually perform poorly](/home/use-tools/types-of-tools#why-llms-perform-poorly-when-calling-http-apis) when calling HTTP APIs. | ||||||
|
||||||
### Tool Context | ||||||
|
||||||
'Tool context' is an object that is passed to a tool as a parameter when the tool is executed. It contains information about the tool call, the user for which the tool is being called, and any secrets the tool requires to run. | ||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,5 @@ | ||
export default { | ||
"tools-overview": "Introduction", | ||
"get-tool-definitions": "Tool formats", | ||
"types-of-tools": "Types of tools", | ||
}; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
--- | ||
title: "Types of Tools" | ||
description: "Learn about LLM-native and API wrapper tools" | ||
--- | ||
|
||
# Types of Tools | ||
|
||
Arcade offers two types of tools: | ||
|
||
- API wrapper tools | ||
- LLM-native tools | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. new name here too! |
||
|
||
The distinction is merely a matter of how they are designed. Both types of tools can be used seamlessly in the same way. There is no difference in their interfaces, the way they are called, or how you interact with them through the Arcade [Dashboard](https://api.arcade.dev/dashboard/) or the Arcade [SDK clients](/home/arcade-clients). | ||
|
||
Before we understand the two types, let's first understand the background for why we need to differentiate between them. | ||
|
||
|
||
## Why LLMs perform poorly when calling HTTP APIs | ||
|
||
Traditionally, the HTTP APIs offered by upstream services such as GitHub, Google, Slack, etc., were designed to be consumed by human software engineers. When we expose such interfaces for LLMs to call as tools, they usually do not perform very well. | ||
|
||
One of the main reasons is that the data model of the HTTP API rarely matches the data model of an AI-powered chat interface. | ||
|
||
For instance, consider the following user prompt: | ||
|
||
> "Send a DM to John asking about a project update" | ||
|
||
The data model mismatches are: | ||
|
||
| Item | Chat interface | Slack HTTP API | | ||
| --- | --- | --- | | ||
| Action | Send message to a user | Send message to a channel | | ||
| Argument | `username = "John"` | `channel_id = ???` | | ||
|
||
|
||
In order to bridge the gap in the data models, the LLM has to make multiple API calls: | ||
|
||
1. Retrieve the current user's Slack ID | ||
2. Browse the list of users to find John's ID | ||
3. Open a direct message between the user and John, and get the channel ID | ||
4. Send the message to the channel | ||
|
||
Even the most powerful LLMs usually perform poorly when they need to reason such complex workflows on the fly, not to mention the increased cost and risk of hallucinations. As a result, AI Agents and chatbots that rely on HTTP APIs often end up being unreliable. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Moving away from generalities like "usually perform poorly", "rarely matches the data model", etc., to specific metrics will help drive the point home. I think it would be really cool to do an eval benchmark for one API wrapper toolkit vs one LLM-native toolkit (for the same upstream provider) to show how much better LLM-native is. We could show a bar chart and start saying things like "performs 76% more poorly", etc. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That would be awesome indeed I'll think of a simple benchmark we can start with |
||
|
||
## LLM-native tools | ||
|
||
Arcade's LLM-native toolkits are designed to match the typical data models expected in AI-powered chat interfaces and are subject to evaluation suites to ensure LLMs can safely use them. | ||
|
||
Following the example above, our Slack toolkit offers the [`Slack.SendMessage`](/toolkits/social-communication/slack#slacksendmessage) tool, which accepts a `username` as argument, matching exactly both the action and argument value expected to be present in the LLM context window. | ||
|
||
When a user says "Send a DM to John asking about a project update", the LLM can directly call the `Slack.SendMessage` tool with the `username` argument, and the tool will take care of the rest. | ||
|
||
LLM-native tools dramatically improve the speed, reliability and cost-effectiveness of AI Agents and chatbots. | ||
|
||
Since they require careful design and evaluation, LLM-native tools take time and effort to build. We understand that your Agent or chatbot project might need capabilities not yet covered by our LLM-native toolkits. For this reason, we also offer low-level API wrapper toolkits. | ||
|
||
## API wrapper tools | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Update the new name here, (and link it) from the glossary |
||
|
||
To provide your Agent or chatbot with more freedom to interact with the upstream services, we offer API wrapper toolkits. | ||
|
||
API wrapper tools are heavily influenced by the original API design. Each tool mirrors one HTTP endpoint. | ||
|
||
Although we redesign the tool name and argument descriptions to make them more suitable for LLMs, API wrapper tools are still not optimized for LLM usage. Also, they are not subject to evaluation suites like LLM-native tools. For those reasons, we recommend thoroughly evaluating each API wrapper tool with your Agents or chatbots before using it in production. | ||
|
||
When your Agent's needs are covered by an LLM-native tool, we recommend using it instead of an API wrapper. Use API wrappers as a complement. Carefully engineer your prompts to ensure your Agent can call them safely. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it can be beneficial to call these something different. I iterated a bit with LLMs, and I like these candidates:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
out of those three I like API Proxy as well