Skip to content

Commit a5fffd4

Browse files
authored
Documents API wrapper vs LLM-native toolkits; includes Slack API wrapper toolkit docs (#435)
* api wrapper docs; slack api wrapper toolkit docs * fix page metadata * fix path to examples; adjust toolkit name * include LLM-native tools and API Wrapper tools in glossary * explain that the differences are merely about how the tools are designed, not how they are used or called * Remove localhost Co-authored-by: Mateo Torres <[email protected]> * rename third-party to upstream * update tools; rename to API Proxy; standardize 'upstream' instead of 'third-party'; use different badge and colors in the catalog * rename to Starter & Optimized; change badges * update badge descriptions; rename to Slack API * fix reference to USER_ID
1 parent 2f568ca commit a5fffd4

File tree

217 files changed

+6972
-78
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

217 files changed

+6972
-78
lines changed

pages/home/glossary.mdx

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,16 @@ Tools are commonly referred to by a qualified name that includes their toolkit.
3030

3131
*Learn more about [tools](/home/build-tools/create-a-toolkit).*
3232

33+
#### Optimized tools
34+
35+
[Optimized tools](/home/use-tools/types-of-tools#optimized-tools) are designed from scratch to provide the best performance for LLMs in terms of speed, reliability, accuracy, and cost-effectiveness.
36+
37+
#### Starter tools
38+
39+
[Starter tools](/home/use-tools/types-of-tools#starter-tools) are designed to mirror the original HTTP API design of the upstream service. They are not optimized for LLM usage and are not subject to evaluation suites. We recommend thoroughly evaluating each Starter tool with your Agents or chatbots before using it in production.
40+
41+
Understand why [LLMs usually perform poorly](/home/use-tools/types-of-tools#why-llms-perform-poorly-when-calling-http-apis) when calling HTTP APIs.
42+
3343
### Tool Context
3444

3545
'Tool context' is an object that is passed to a tool as a parameter when the tool is executed. It contains information about the tool call, the user for which the tool is being called, and any secrets the tool requires to run.

pages/home/use-tools/_meta.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
export default {
22
"tools-overview": "Introduction",
33
"get-tool-definitions": "Tool formats",
4+
"types-of-tools": "Types of tools",
45
};
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
---
2+
title: "Types of Tools"
3+
description: "Learn about Optimized and Starter tools"
4+
---
5+
6+
# Types of Tools
7+
8+
Arcade offers two types of tools:
9+
10+
- Starter tools
11+
- Optimized tools
12+
13+
The distinction is merely a matter of how they are designed. Both types of tools can be used seamlessly in the same way. There is no difference in their interfaces, the way they are called, or how you interact with them through the Arcade [Dashboard](https://api.arcade.dev/dashboard/) or the Arcade [SDK clients](/home/arcade-clients).
14+
15+
Before we understand the two types, let's first understand the background for why we need to differentiate between them.
16+
17+
18+
## Why LLMs perform poorly when calling HTTP APIs
19+
20+
Traditionally, the HTTP APIs offered by upstream services such as GitHub, Google, Slack, etc., were designed to be consumed by human software engineers. When we expose such interfaces for LLMs to call as tools, they usually do not perform very well.
21+
22+
One of the main reasons is that the data model of the HTTP API rarely matches the data model of an AI-powered chat interface.
23+
24+
For instance, consider the following user prompt:
25+
26+
> "Send a DM to John asking about a project update"
27+
28+
The data model mismatches are:
29+
30+
| Dimension | Chat interface | Slack HTTP API |
31+
| --- | --- | --- |
32+
| Action | Send message to a <b><i>person</i></b> | Send message to a <b><i>channel</i></b> |
33+
| Argument | `username = "John"` | `channel_id = ???` |
34+
35+
36+
In order to bridge the gap in the data models, the LLM has to make multiple API calls:
37+
38+
1. Retrieve the current user's Slack ID
39+
2. Browse the list of users to find John's ID
40+
3. Open a DM (direct message) channel between the user and John, and get this channel's ID
41+
4. Send the message to the channel
42+
43+
Even the most powerful LLMs usually perform poorly when they need to reason such complex workflows on the fly, not to mention the increased cost and risk of hallucinations. As a result, AI Agents and chatbots that rely on HTTP APIs often end up being unreliable.
44+
45+
## Optimized tools
46+
47+
Arcade's Optimized toolkits are designed to match the typical data models expected in AI-powered chat interfaces and are subject to evaluation suites to ensure LLMs can safely use them.
48+
49+
Following the example above, our Slack toolkit offers the [`Slack.SendMessage`](/toolkits/social-communication/slack#slacksendmessage) tool, which accepts a `username` as argument, matching exactly both the action and argument value expected to be present in the LLM context window.
50+
51+
When a user says "Send a DM to John asking about a project update", the LLM can directly call the `Slack.SendMessage` tool with the `username` argument, and the tool will take care of the rest.
52+
53+
Optimized tools dramatically improve the speed, reliability and cost-effectiveness of AI Agents and chatbots.
54+
55+
Since they require careful design and evaluation, Optimized tools take time and effort to build. We understand that your Agent or chatbot project might need capabilities not yet covered by our Optimized toolkits. For this reason, we also offer low-level Starter toolkits.
56+
57+
## Starter tools
58+
59+
To provide your Agent or chatbot with more freedom to interact with the upstream services, we offer Starter toolkits.
60+
61+
Starter tools are heavily influenced by the original API design. Each tool mirrors one HTTP endpoint.
62+
63+
Although we redesign the tool name and argument descriptions to make them more suitable for LLMs, Starter tools are still not optimized for LLM usage. Also, they are not subject to evaluation suites like Optimized tools. For those reasons, we recommend thoroughly evaluating each Starter tool with your Agents or chatbots before using it in production.
64+
65+
When your Agent's needs are covered by an Optimized tool, we recommend using it instead of a Starter one. Use Starter tools as a complement. Carefully engineer your prompts to ensure your Agent can call them safely.

0 commit comments

Comments
 (0)