Skip to content

Commit fef739a

Browse files
tylerslatontylerslaton
committed
feat(dojo): add backend_tool_rendering feature
Signed-off-by: Tyler Slaton <[email protected]>
1 parent 6ab8337 commit fef739a

File tree

23 files changed

+950
-73
lines changed

23 files changed

+950
-73
lines changed

‎test-results/.last-run.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"status": "failed",
3+
"failedTests": []
4+
}

‎typescript-sdk/apps/dojo/e2e/tests/mastraAgentLocalTests/backendToolRenderingPage.spec.ts‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
import { test, expect } from "@playwright/test";
2+
3+
test("[MastraAgentLocal] Backend Tool Rendering displays weather cards", async ({ page }) => {
4+
// Set shorter default timeout for this test
5+
test.setTimeout(30000); // 30 seconds total
6+
7+
await page.goto("/mastra-agent-local/feature/backend_tool_rendering");
8+
9+
// Verify suggestion buttons are visible
10+
await expect(page.getByRole("button", { name: "Weather in San Francisco" })).toBeVisible({
11+
timeout: 5000,
12+
});
13+
14+
// Click first suggestion and verify weather card appears
15+
await page.getByRole("button", { name: "Weather in San Francisco" }).click();
16+
17+
// Wait for either test ID or fallback to "Current Weather" text
18+
const weatherCard = page.getByTestId("weather-card");
19+
const currentWeatherText = page.getByText("Current Weather");
20+
21+
// Try test ID first, fallback to text
22+
try {
23+
await expect(weatherCard).toBeVisible({ timeout: 10000 });
24+
} catch (e) {
25+
// Fallback to checking for "Current Weather" text
26+
await expect(currentWeatherText.first()).toBeVisible({ timeout: 10000 });
27+
}
28+
29+
// Verify weather content is present (use flexible selectors)
30+
const hasHumidity = await page
31+
.getByText("Humidity")
32+
.isVisible()
33+
.catch(() => false);
34+
const hasWind = await page
35+
.getByText("Wind")
36+
.isVisible()
37+
.catch(() => false);
38+
const hasCityName = await page
39+
.locator("h3")
40+
.filter({ hasText: /San Francisco/i })
41+
.isVisible()
42+
.catch(() => false);
43+
44+
// At least one of these should be true
45+
expect(hasHumidity || hasWind || hasCityName).toBeTruthy();
46+
47+
// Click second suggestion
48+
await page.getByRole("button", { name: "Weather in New York" }).click();
49+
await page.waitForTimeout(2000);
50+
51+
// Verify at least one weather-related element is still visible
52+
const weatherElements = await page.getByText(/Weather|Humidity|Wind|Temperature/i).count();
53+
expect(weatherElements).toBeGreaterThan(0);
54+
});

‎typescript-sdk/apps/dojo/e2e/tests/mastraTests/backendToolRenderingPage.spec.ts‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
import { test, expect } from "@playwright/test";
2+
3+
test("[MastraAgentLocal] Backend Tool Rendering displays weather cards", async ({ page }) => {
4+
// Set shorter default timeout for this test
5+
test.setTimeout(30000); // 30 seconds total
6+
7+
await page.goto("/mastra/feature/backend_tool_rendering");
8+
9+
// Verify suggestion buttons are visible
10+
await expect(page.getByRole("button", { name: "Weather in San Francisco" })).toBeVisible({
11+
timeout: 5000,
12+
});
13+
14+
// Click first suggestion and verify weather card appears
15+
await page.getByRole("button", { name: "Weather in San Francisco" }).click();
16+
17+
// Wait for either test ID or fallback to "Current Weather" text
18+
const weatherCard = page.getByTestId("weather-card");
19+
const currentWeatherText = page.getByText("Current Weather");
20+
21+
// Try test ID first, fallback to text
22+
try {
23+
await expect(weatherCard).toBeVisible({ timeout: 10000 });
24+
} catch (e) {
25+
// Fallback to checking for "Current Weather" text
26+
await expect(currentWeatherText.first()).toBeVisible({ timeout: 10000 });
27+
}
28+
29+
// Verify weather content is present (use flexible selectors)
30+
const hasHumidity = await page
31+
.getByText("Humidity")
32+
.isVisible()
33+
.catch(() => false);
34+
const hasWind = await page
35+
.getByText("Wind")
36+
.isVisible()
37+
.catch(() => false);
38+
const hasCityName = await page
39+
.locator("h3")
40+
.filter({ hasText: /San Francisco/i })
41+
.isVisible()
42+
.catch(() => false);
43+
44+
// At least one of these should be true
45+
expect(hasHumidity || hasWind || hasCityName).toBeTruthy();
46+
47+
// Click second suggestion
48+
await page.getByRole("button", { name: "Weather in New York" }).click();
49+
await page.waitForTimeout(2000);
50+
51+
// Verify at least one weather-related element is still visible
52+
const weatherElements = await page.getByText(/Weather|Humidity|Wind|Temperature/i).count();
53+
expect(weatherElements).toBeGreaterThan(0);
54+
});

‎typescript-sdk/apps/dojo/e2e/tests/serverStarterAllFeaturesTests/backendToolRenderingPage.spec.ts‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
import { test, expect } from "@playwright/test";
2+
3+
test("[MastraAgentLocal] Backend Tool Rendering displays weather cards", async ({ page }) => {
4+
// Set shorter default timeout for this test
5+
test.setTimeout(30000); // 30 seconds total
6+
7+
await page.goto("/server-starter-all-features/feature/agentic_generative_ui");
8+
9+
// Verify suggestion buttons are visible
10+
await expect(page.getByRole("button", { name: "Weather in San Francisco" })).toBeVisible({
11+
timeout: 5000,
12+
});
13+
14+
// Click first suggestion and verify weather card appears
15+
await page.getByRole("button", { name: "Weather in San Francisco" }).click();
16+
17+
// Wait for either test ID or fallback to "Current Weather" text
18+
const weatherCard = page.getByTestId("weather-card");
19+
const currentWeatherText = page.getByText("Current Weather");
20+
21+
// Try test ID first, fallback to text
22+
try {
23+
await expect(weatherCard).toBeVisible({ timeout: 10000 });
24+
} catch (e) {
25+
// Fallback to checking for "Current Weather" text
26+
await expect(currentWeatherText.first()).toBeVisible({ timeout: 10000 });
27+
}
28+
29+
// Verify weather content is present (use flexible selectors)
30+
const hasHumidity = await page
31+
.getByText("Humidity")
32+
.isVisible()
33+
.catch(() => false);
34+
const hasWind = await page
35+
.getByText("Wind")
36+
.isVisible()
37+
.catch(() => false);
38+
const hasCityName = await page
39+
.locator("h3")
40+
.filter({ hasText: /San Francisco/i })
41+
.isVisible()
42+
.catch(() => false);
43+
44+
// At least one of these should be true
45+
expect(hasHumidity || hasWind || hasCityName).toBeTruthy();
46+
47+
// Click second suggestion
48+
await page.getByRole("button", { name: "Weather in New York" }).click();
49+
await page.waitForTimeout(2000);
50+
51+
// Verify at least one weather-related element is still visible
52+
const weatherElements = await page.getByText(/Weather|Humidity|Wind|Temperature/i).count();
53+
expect(weatherElements).toBeGreaterThan(0);
54+
});

‎typescript-sdk/apps/dojo/src/agents.ts‎

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,8 +68,12 @@ export const agentsIntegrations: AgentIntegrationConfig[] = [
6868
agents: async () => {
6969
return {
7070
agentic_chat: new ADKAgent({ url: `${envVars.adkMiddlewareUrl}/chat` }),
71-
tool_based_generative_ui: new ADKAgent({ url: `${envVars.adkMiddlewareUrl}/adk-tool-based-generative-ui` }),
72-
human_in_the_loop: new ADKAgent({ url: `${envVars.adkMiddlewareUrl}/adk-human-in-loop-agent` }),
71+
tool_based_generative_ui: new ADKAgent({
72+
url: `${envVars.adkMiddlewareUrl}/adk-tool-based-generative-ui`,
73+
}),
74+
human_in_the_loop: new ADKAgent({
75+
url: `${envVars.adkMiddlewareUrl}/adk-human-in-loop-agent`,
76+
}),
7377
shared_state: new ADKAgent({ url: `${envVars.adkMiddlewareUrl}/adk-shared-state-agent` }),
7478
// predictive_state_updates: new ADKAgent({ url: `${envVars.adkMiddlewareUrl}/adk-predictive-state-agent` }),
7579
};
@@ -82,6 +86,9 @@ export const agentsIntegrations: AgentIntegrationConfig[] = [
8286
agentic_chat: new ServerStarterAllFeaturesAgent({
8387
url: `${envVars.serverStarterAllFeaturesUrl}/agentic_chat`,
8488
}),
89+
backend_tool_rendering: new ServerStarterAllFeaturesAgent({
90+
url: `${envVars.serverStarterAllFeaturesUrl}/backend_tool_rendering`,
91+
}),
8592
human_in_the_loop: new ServerStarterAllFeaturesAgent({
8693
url: `${envVars.serverStarterAllFeaturesUrl}/human_in_the_loop`,
8794
}),
@@ -135,6 +142,10 @@ export const agentsIntegrations: AgentIntegrationConfig[] = [
135142
deploymentUrl: envVars.langgraphPythonUrl,
136143
graphId: "agentic_chat",
137144
}),
145+
backend_tool_rendering: new LangGraphAgent({
146+
deploymentUrl: envVars.langgraphPythonUrl,
147+
graphId: "backend_tool_rendering",
148+
}),
138149
agentic_generative_ui: new LangGraphAgent({
139150
deploymentUrl: envVars.langgraphPythonUrl,
140151
graphId: "agentic_generative_ui",
@@ -172,6 +183,9 @@ export const agentsIntegrations: AgentIntegrationConfig[] = [
172183
agentic_chat: new LangGraphHttpAgent({
173184
url: `${envVars.langgraphFastApiUrl}/agent/agentic_chat`,
174185
}),
186+
backend_tool_rendering: new LangGraphHttpAgent({
187+
url: `${envVars.langgraphFastApiUrl}/agent/backend_tool_rendering`,
188+
}),
175189
agentic_generative_ui: new LangGraphHttpAgent({
176190
url: `${envVars.langgraphFastApiUrl}/agent/agentic_generative_ui`,
177191
}),
@@ -204,6 +218,10 @@ export const agentsIntegrations: AgentIntegrationConfig[] = [
204218
deploymentUrl: envVars.langgraphTypescriptUrl,
205219
graphId: "agentic_chat",
206220
}),
221+
// agentic_chat_reasoning: new LangGraphAgent({
222+
// deploymentUrl: envVars.langgraphTypescriptUrl,
223+
// graphId: "agentic_chat_reasoning",
224+
// }),
207225
agentic_generative_ui: new LangGraphAgent({
208226
deploymentUrl: envVars.langgraphTypescriptUrl,
209227
graphId: "agentic_generative_ui",
@@ -292,7 +310,11 @@ export const agentsIntegrations: AgentIntegrationConfig[] = [
292310
id: "a2a",
293311
agents: async () => {
294312
// A2A agents: building management, finance, it agents
295-
const agentUrls = [envVars.a2aMiddlewareBuildingsManagementUrl, envVars.a2aMiddlewareFinanceUrl, envVars.a2aMiddlewareItUrl];
313+
const agentUrls = [
314+
envVars.a2aMiddlewareBuildingsManagementUrl,
315+
envVars.a2aMiddlewareFinanceUrl,
316+
envVars.a2aMiddlewareItUrl,
317+
];
296318
// AGUI orchestration/routing agent
297319
const orchestrationAgent = new HttpAgent({
298320
url: envVars.a2aMiddlewareOrchestratorUrl,

‎typescript-sdk/apps/dojo/src/app/[integrationId]/feature/agentic_chat/page.tsx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ const Chat = () => {
5757
>
5858
<div className="h-full w-full md:w-8/10 md:h-8/10 rounded-lg">
5959
<CopilotChat
60-
className="h-full rounded-2xl"
60+
className="h-full rounded-2xl max-w-6xl mx-auto"
6161
labels={{ initial: "Hi, I'm an agent. Want to chat?" }}
6262
suggestions={[
6363
{

‎typescript-sdk/apps/dojo/src/app/[integrationId]/feature/agentic_generative_ui/page.tsx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -215,7 +215,7 @@ const Chat = () => {
215215
<div className="flex justify-center items-center h-full w-full">
216216
<div className="h-full w-full md:w-8/10 md:h-8/10 rounded-lg">
217217
<CopilotChat
218-
className="h-full rounded-2xl"
218+
className="h-full rounded-2xl max-w-6xl mx-auto"
219219
labels={{
220220
initial:
221221
"Hi, I'm an agent! I can help you with anything you need and will show you progress as I work. What can I do for you?",

‎typescript-sdk/apps/dojo/src/app/[integrationId]/feature/backend_tool_rendering/README.mdx‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# 🤖 Agentic Chat with Frontend Tools
2+
3+
## What This Demo Shows
4+
5+
This demo showcases CopilotKit's **agentic chat** capabilities with **frontend
6+
tool integration**:
7+
8+
1. **Natural Conversation**: Chat with your Copilot in a familiar chat interface
9+
2. **Frontend Tool Execution**: The Copilot can directly interacts with your UI
10+
by calling frontend functions
11+
3. **Seamless Integration**: Tools defined in the frontend and automatically
12+
discovered and made available to the agent
13+
14+
## How to Interact
15+
16+
Try asking your Copilot to:
17+
18+
- "Can you change the background color to something more vibrant?"
19+
- "Make the background a blue to purple gradient"
20+
- "Set the background to a sunset-themed gradient"
21+
- "Change it back to a simple light color"
22+
23+
You can also chat about other topics - the agent will respond conversationally
24+
while having the ability to use your UI tools when appropriate.
25+
26+
## ✨ Frontend Tool Integration in Action
27+
28+
**What's happening technically:**
29+
30+
- The React component defines a frontend function using `useCopilotAction`
31+
- CopilotKit automatically exposes this function to the agent
32+
- When you make a request, the agent determines whether to use the tool
33+
- The agent calls the function with the appropriate parameters
34+
- The UI immediately updates in response
35+
36+
**What you'll see in this demo:**
37+
38+
- The Copilot understands requests to change the background
39+
- It generates CSS values for colors and gradients
40+
- When it calls the tool, the background changes instantly
41+
- The agent provides a conversational response about the changes it made
42+
43+
This technique of exposing frontend functions to your Copilot can be extended to
44+
any UI manipulation you want to enable, from theme changes to data filtering,
45+
navigation, or complex UI state management!

0 commit comments

Comments
 (0)