feat(ui): initial canvas functionality #1667

kapetr · 2025-12-02T13:55:47Z

Summary

Adds initial Canvas functionality to the UI

Linked Issues

Refs #168

Documentation

No Docs Needed:

Breaking changes

TS SDK

handleInputRequired -> resolveUserMetadata
InputRequiredResponses -> UserMetadataInputs

gemini-code-assist · 2025-12-02T13:56:43Z

Summary of Changes

Hello @kapetr, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request delivers the initial implementation of Canvas functionality, a significant new feature that allows for interactive content within the user interface. It encompasses updates across the Python and TypeScript SDKs to integrate the Canvas extension, alongside the development of new UI components for rendering, managing, and interacting with Canvas-based artifacts. Users can now select text within a Canvas and submit edit requests to the agent, enhancing the interactive experience.

Highlights

Initial Canvas Functionality: Introduced the foundational Canvas feature to the UI, enabling agents to interact with and modify rich content directly within the chat interface.
SDK Updates: Updated both Python and TypeScript SDKs to support the new Canvas extension, including new examples for code and recipe agents.
UI Components and Interaction: Added new UI components for displaying Canvas artifacts, managing their state, and allowing user interaction through text selection and editing requests.
Markdown Enhancements: Implemented markdown rendering enhancements to track source positions of content and enable a floating toolbar for selected text within Canvas artifacts.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request introduces initial Canvas functionality to the UI, which involves significant changes across both Python SDK examples and the TypeScript UI. Key updates include new UI components for displaying and interacting with canvas artifacts, a mechanism to map DOM selections back to markdown source for editing, and integration of canvas-specific message parts and API extensions. The changes are generally well-structured and introduce a powerful new feature. However, there are a few areas that require attention regarding robustness, consistency, and potential breaking changes in the TypeScript SDK.

apps/agentstack-sdk-ts/src/client/a2a/extensions/handle-agent-card.ts

apps/agentstack-ui/src/modules/runs/contexts/agent-demands/build-fulfillments.ts

apps/agentstack-sdk-py/examples/canvas_ui_code_agent.py

apps/agentstack-sdk-py/examples/canvas_ui_test_agent.py

PetrBulanek · 2025-12-05T09:00:40Z

apps/agentstack-sdk-ts/src/client/a2a/extensions/ui/canvas.ts

+
+const URI = 'https://a2a-extensions.agentstack.beeai.dev/ui/canvas/v1';
+
+const schema = z.null();


This feels weird to me. Shouldn't we rather put an empty object there? Maybe there will be some demands in the future, so let's prepare for that right away?

I agree and it would simplify things on the UI. What do you think @tomkis ?

PetrBulanek · 2025-12-05T09:02:53Z

apps/agentstack-sdk-ts/src/client/a2a/extensions/utils.ts

Why this change? :-)

So a demand with empty parameters can be distinguished from a non-existent demand. But I completely agree with your concern - we should rethink how this is handled.

PetrBulanek · 2025-12-05T09:05:16Z

apps/agentstack-sdk-ts/src/client/a2a/extensions/handle-agent-card.ts

+    if (canvasDemands !== undefined) {
+      const canvasEditRequest = fulfillments.canvasEditRequest();
+      if (canvasEditRequest) {
+        fulfilledMetadata = fulfillCanvasDemand(fulfilledMetadata, canvasEditRequest);
+      }
+    }
+
    const oauthRedirectUri = fulfillments.oauthRedirectUri();


Why are you checking undefined directly here? I would probably try to make it all as similar as possible to the other functions (with regard to parameters, return values, etc.). Maybe in the future some mechanism will be created so that all this doesn't have to be called manually. :-)

The same reason as the previous comment.

PetrBulanek · 2025-12-05T09:11:00Z

apps/agentstack-ui/src/api/a2a/client.ts

-            const parts = handleArtifactUpdate(event);
+            const parts = handleArtifactUpdate(event, demands.canvasDemands !== undefined);


Is this a good solution? Theoretically, it could just be a matter of returning the file and not the canvas itself. Shouldn't this be differentiated more at the metadata level? I'd be happy to talk about it some more. This seems fragile to me.

See above comments. Not sure i understand the comment with the file (it can still return FilePart). This essentially checks if the agent can handle canvas edit requests, meaning it will add ArtifactPart and render Canvas UI.
But I'll be happy to discuss this, it's really an initial version!

apps/agentstack-ui/src/api/a2a/client.ts

apps/agentstack-ui/src/modules/canvas/markdown/hooks/useMarkdownSelectionDialog.ts

apps/agentstack-ui/src/modules/canvas/markdown/hooks/useTextSelection.ts

apps/agentstack-ui/src/modules/canvas/markdown/utils/mapDOMSelectionToMarkdown.ts

PetrBulanek · 2025-12-07T11:54:23Z

apps/agentstack-ui/src/modules/canvas/utils.ts

+import { findWithIndex } from '#utils/helpers.ts';
+import { toMarkdownArtifact } from '#utils/markdown.ts';
+
+export function processMessageArtifactPart(


Perhaps it would be better if the function returned parts and the array mutation took place outside the function. This is how it is handled in other cases in addMessagePart fn. I don't see any actual problem there, but it seems a little clearer to me.

That would basically mean removing this function and copying contents into the addMessagePart. I'd prefer that as well, but it would clutter addMessagePart quite a bit.

Got it. I just found the function a little difficult to read. Maybe it's primarily because of the long if conditions. But I guess it's not such a big deal. So let's leave it as it is. :-)

Signed-off-by: Petr Kadlec <[email protected]>

…wn source Signed-off-by: Petr Kadlec <[email protected]>

Signed-off-by: Petr Kadlec <[email protected]>

Signed-off-by: Petr Kadlec <[email protected]> feat(ui): finalize canvas, improve selection mapping Signed-off-by: Petr Kadlec <[email protected]> Signed-off-by: Petr Kadlec <[email protected]> Signed-off-by: Petr Kadlec <[email protected]> Signed-off-by: Petr Kadlec <[email protected]>

Signed-off-by: Petr Kadlec <[email protected]>

kapetr · 2025-12-09T13:46:01Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces initial Canvas functionality to the UI, enabling users to interact with artifacts displayed in a canvas-like view. Key changes include new Python SDK examples for canvas agents, TypeScript SDK updates for canvas extensions, and significant UI components for rendering and interacting with canvas artifacts. The UI now supports selecting text within markdown content to initiate edit requests, and a new context (CanvasProvider) manages active artifacts. Overall, the changes are well-structured and integrate the new feature effectively. However, there are a few areas that could be improved for robustness and clarity, particularly regarding state management and selection logic.

apps/agentstack-ui/src/modules/canvas/contexts/CanvasProvider.tsx

apps/agentstack-ui/src/modules/canvas/markdown/utils/mapDOMSelectionToMarkdown.ts

gemini-code-assist · 2025-12-09T13:51:10Z

apps/agentstack-sdk-py/src/agentstack_sdk/a2a/extensions/ui/canvas.py

+        if message.metadata and self.spec.URI in message.metadata and message.parts:
+            message.parts = [part for part in message.parts if not isinstance(part.root, TextPart)]


Modifying message.parts in place within handle_incoming_message can lead to unexpected side effects if the A2AMessage object is intended to be immutable or if other parts of the system rely on its original state. It's generally safer to create a new list of parts rather than mutating the incoming message directly.

Suggested change

if message.metadata and self.spec.URI in message.metadata and message.parts:

message.parts = [part for part in message.parts if not isinstance(part.root, TextPart)]

if message.metadata and self.spec.URI in message.metadata and message.parts:

# Create a new list of parts to avoid mutating the original message object

message.parts = [part for part in message.parts if not isinstance(part.root, TextPart)]

apps/agentstack-ui/src/modules/canvas/components/Canvas.tsx

tomkis · 2025-12-10T09:56:11Z

apps/agentstack-sdk-ts/src/client/a2a/extensions/ui/canvas.ts

+  artifact_id: z.string(),
+});
+
+export type CanvasFulfillments = z.infer<typeof responseSchema>;


Lets rename this, it's no longer fullfilment.

tomkis · 2025-12-10T10:04:33Z

apps/agentstack-sdk-ts/src/client/a2a/extensions/handle-agent-card.ts

      };
    }

+    const canvasEditRequest = fulfillments.canvasEditRequest();


This is wrong. Canvas Edit Request is not fullfilemnt.

Signed-off-by: Petr Kadlec <[email protected]>

PetrBulanek · 2025-12-10T19:50:35Z

apps/agentstack-ui/src/api/a2a/client.ts

-      const inputRequiredMetadata = await resolveInputRequiredMetadata(responses);
+      const uiExtensionMetadata = await resolveUiExtensionMetadata(uiExtensionInputs);
+
+      console.log({ uiExtensionInputs, uiExtensionMetadata });


Forgotten console.log

Signed-off-by: Petr Kadlec <[email protected]>

kapetr · 2025-12-11T08:48:49Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces initial canvas functionality, a significant new feature enabling users to interact with agent-generated artifacts like code or recipes in a dedicated UI panel. The implementation spans the Python and TypeScript SDKs and the UI application, featuring artifact streaming, an editing interface with text selection, and markdown-to-DOM position mapping. The changes are extensive and well-structured. I've identified a high-severity bug in the UI's artifact streaming logic that could cause data duplication, and two medium-severity issues in the Python example files related to unsafe array access that could lead to crashes. Addressing these points will enhance the feature's robustness.

apps/agentstack-ui/src/modules/canvas/utils.ts

apps/agentstack-sdk-py/examples/canvas_ui_code_agent.py

apps/agentstack-sdk-py/examples/canvas_ui_test_agent.py

Signed-off-by: Petr Kadlec <[email protected]>

kapetr · 2025-12-11T14:29:45Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces significant new functionality for a 'Canvas' UI, allowing users to select and request edits on artifacts generated by an agent. The changes span across the Python SDK, the TypeScript SDK, and the UI application, including new components, hooks, and context providers. The implementation is quite comprehensive, covering everything from the backend agent extension to the frontend selection mechanism with a floating toolbar.

My review focuses on improving robustness, performance, and correctness in the new example files and core logic. I've identified a few areas for improvement:

In the Python example agents, there are opportunities to make artifact part access safer and to optimize regex compilation.
In the Python canvas extension, a type check appears to be incorrect, which could lead to bugs.
In the UI, there's a minor bug in how artifact content is constructed and an accessibility issue in a new form.
The new documentation for the canvas feature contains a couple of errors in its code examples that could mislead developers.

Overall, this is a well-executed feature addition. The use of modern browser APIs like CSS Custom Highlights is a nice touch. Addressing the feedback will make the implementation more robust and the documentation more accurate.

apps/agentstack-sdk-py/src/agentstack_sdk/a2a/extensions/ui/canvas.py

apps/agentstack-ui/src/modules/canvas/components/Canvas.tsx

docs/agent-development/canvas.mdx

apps/agentstack-sdk-py/examples/canvas_ui_code_agent.py

apps/agentstack-sdk-py/examples/canvas_ui_test_agent.py

apps/agentstack-ui/src/modules/canvas/markdown/CanvasEditForm.tsx

apps/agentstack-ui/src/modules/messages/utils.ts

docs/agent-development/canvas.mdx

Signed-off-by: Petr Kadlec <[email protected]>

dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Dec 2, 2025

gemini-code-assist bot reviewed Dec 2, 2025

View reviewed changes

kapetr requested a review from PetrBulanek December 2, 2025 15:21

kapetr force-pushed the feat/168-open-canvas-functionality branch from 5220d2f to b2ab5ef Compare December 3, 2025 10:47

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Dec 3, 2025

PetrBulanek requested changes Dec 7, 2025

View reviewed changes

kapetr added 7 commits December 8, 2025 10:31

feat(ui): artifacts handling for the canvas agent

d4f5e0c

Signed-off-by: Petr Kadlec <[email protected]>

feat(ui): process artifacts and display in GUI

bd3ed9c

Signed-off-by: Petr Kadlec <[email protected]>

feat(ui): add canvas edit toolbar, map text selection range to markdo…

dc54eca

…wn source Signed-off-by: Petr Kadlec <[email protected]>

feat(ui): submit canvas edit request

a950788

Signed-off-by: Petr Kadlec <[email protected]>

feat(ui): add streaming support to canvas artifacts

7130ead

Signed-off-by: Petr Kadlec <[email protected]>

fixup! review comments

c568163

Signed-off-by: Petr Kadlec <[email protected]>

kapetr force-pushed the feat/168-open-canvas-functionality branch from b2ab5ef to c568163 Compare December 8, 2025 15:45

fix(ui): rework canvas extension handling

41aa5b8

Signed-off-by: Petr Kadlec <[email protected]>

kapetr requested a review from tomkis December 9, 2025 13:25

gemini-code-assist bot reviewed Dec 9, 2025

View reviewed changes

kapetr requested a review from PetrBulanek December 10, 2025 07:48

tomkis reviewed Dec 10, 2025

View reviewed changes

PetrBulanek approved these changes Dec 10, 2025

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 10, 2025

fixup! refactor canvas edit request metadata handling

be90b89

Signed-off-by: Petr Kadlec <[email protected]>

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Dec 10, 2025

PetrBulanek requested changes Dec 10, 2025

View reviewed changes

dosubot bot removed the lgtm This PR has been approved by a maintainer label Dec 10, 2025

fixup! remove forgotten log

c6c420b

Signed-off-by: Petr Kadlec <[email protected]>

gemini-code-assist bot reviewed Dec 11, 2025

View reviewed changes

apps/agentstack-ui/src/modules/canvas/utils.ts Show resolved Hide resolved

apps/agentstack-sdk-py/examples/canvas_ui_code_agent.py Show resolved Hide resolved

apps/agentstack-sdk-py/examples/canvas_ui_test_agent.py Show resolved Hide resolved

fixup! rename user metadata handler

cbc46fe

Signed-off-by: Petr Kadlec <[email protected]>

kapetr requested a review from tomkis December 11, 2025 10:27

kapetr added the breaking-change label Dec 11, 2025

kapetr requested a review from PetrBulanek December 11, 2025 12:13

tomkis approved these changes Dec 11, 2025

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 11, 2025

PetrBulanek approved these changes Dec 11, 2025

View reviewed changes

kapetr added 2 commits December 11, 2025 15:08

chore: add canvas docs

6847184

Signed-off-by: Petr Kadlec <[email protected]>

Merge branch 'main' into feat/168-open-canvas-functionality

6544007

Signed-off-by: Petr Kadlec <[email protected]>

gemini-code-assist bot reviewed Dec 11, 2025

View reviewed changes

fixup! review comments

4650c02

Signed-off-by: Petr Kadlec <[email protected]>

kapetr merged commit 965df7f into main Dec 11, 2025
10 checks passed

kapetr deleted the feat/168-open-canvas-functionality branch December 11, 2025 14:56


		const URI = 'https://a2a-extensions.agentstack.beeai.dev/ui/canvas/v1';

		const schema = z.null();

		const parts = handleArtifactUpdate(event);
		const parts = handleArtifactUpdate(event, demands.canvasDemands !== undefined);

		if message.metadata and self.spec.URI in message.metadata and message.parts:
		message.parts = [part for part in message.parts if not isinstance(part.root, TextPart)]

feat(ui): initial canvas functionality #1667

feat(ui): initial canvas functionality #1667

Uh oh!

Conversation

kapetr commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Linked Issues

Documentation

Breaking changes

TS SDK

Uh oh!

gemini-code-assist bot commented Dec 2, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kapetr commented Dec 9, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kapetr commented Dec 11, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kapetr commented Dec 2, 2025 •

edited

Loading