feat: added AI image tools for view and upload to project operations #2939
Conversation
|
@iNerdStack is attempting to deploy a commit to the Onlook Team on Vercel. A member of the Team first needs to authorize it. |
|
Note Other AI code review bot(s) detectedCodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review. WalkthroughAdds two image-focused tools (ViewImageTool, UploadImageTool), registers them in tool registries and exports, inlines available-images guidance into AI prompts, and updates chat UI and image contexts to render and reference image inputs/outputs with explicit image IDs. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor User
participant ChatUI as Chat UI
participant Agent as Agent
participant Tools as Tools Registry
participant View as ViewImageTool
participant Upload as UploadImageTool
participant Engine as EditorEngine
participant FS as File System
User->>ChatUI: Sends message (may reference image_id)
ChatUI->>Agent: Prompt (includes <available-images/> guidance)
Agent->>Tools: Select tool by name
alt view_image
Agent->>View: handle({ image_id })
View->>Engine: Read chat image contexts (match image_id)
View-->>Agent: { image:{mimeType,data}, message }
Agent-->>ChatUI: Tool output
ChatUI-->>User: Renders "View Image" block
else upload_image
Agent->>Upload: handle({ image_id, destination_path, filename, branchId })
Upload->>Engine: Resolve sandbox & read image context
Upload->>FS: Write binary file to sandbox (uses mime-lite for extension)
Upload->>Engine: Trigger image indexing/scan
Upload-->>Agent: "Uploaded to <path>"
Agent-->>ChatUI: Tool output
ChatUI-->>User: Renders "Upload Image" block
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
...ct/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
Show resolved
Hide resolved
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (4)
packages/ai/src/tools/toolset.ts (1)
69-70: Consider adding VIEW_IMAGE_TOOL to ASK_TOOL_SET.The
view_imagetool is read-only (retrieves image data for analysis without modifying the project), which aligns with the purpose of ASK mode. Currently, it's only available in BUILD mode.If users should be able to view/analyze images in ASK mode, add
VIEW_IMAGE_TOOLtoASK_TOOL_SET:export const ASK_TOOL_SET: ToolSet = { [LIST_FILES_TOOL_NAME]: listFilesTool, [READ_FILE_TOOL_NAME]: readFileTool, [BASH_READ_TOOL_NAME]: bashReadTool, [ONLOOK_INSTRUCTIONS_TOOL_NAME]: onlookInstructionsTool, [READ_STYLE_GUIDE_TOOL_NAME]: readStyleGuideTool, [LIST_BRANCHES_TOOL_NAME]: listBranchesTool, [SCRAPE_URL_TOOL_NAME]: scrapeUrlTool, [WEB_SEARCH_TOOL_NAME]: webSearchTool, [GLOB_TOOL_NAME]: globTool, [GREP_TOOL_NAME]: grepTool, + [VIEW_IMAGE_TOOL_NAME]: viewImageTool, };Note:
UPLOAD_IMAGE_TOOLcorrectly belongs only in BUILD mode since it modifies the project.apps/web/client/src/components/tools/handlers/edit.ts (3)
175-222: Simplify the lookup logic to reduce duplication.The function contains duplicate fallback logic for index-based retrieval. The initial
findat lines 179-188 already attempts index matching, but if it fails, lines 190-207 repeat similar logic.Consider refactoring to eliminate the duplication:
export async function handleViewImageTool(args: z.infer<typeof VIEW_IMAGE_TOOL_PARAMETERS>, editorEngine: EditorEngine): Promise<{ image: { mimeType: string; data: string }; message: string }> { try { - // Find the image in the chat context by reference const context = editorEngine.chat.context.context; - const imageContext = context.find((ctx) => { + const imageContexts = context.filter(ctx => ctx.type === MessageContextType.IMAGE); + + // Try index-based lookup first + const indexMatch = args.image_reference.match(/^\d+$/); + if (indexMatch) { + const index = parseInt(indexMatch[0]) - 1; + if (index >= 0 && index < imageContexts.length) { + const foundImage = imageContexts[index]; + return { + image: { + mimeType: foundImage.mimeType, + data: foundImage.content, + }, + message: `Retrieved image "${foundImage.displayName}" for analysis.`, + }; + } + } + + // Try name-based lookup + const imageContext = imageContexts.find((ctx) => { - if (ctx.type !== MessageContextType.IMAGE) { - return false; - } - // Try to match by display name, index number, or description const ref = args.image_reference.toLowerCase(); return ctx.displayName.toLowerCase().includes(ref) || - ref.includes(ctx.displayName.toLowerCase()) || - ref.match(/^\d+$/) && context.filter(c => c.type === MessageContextType.IMAGE)[parseInt(ref) - 1] === ctx; + ref.includes(ctx.displayName.toLowerCase()); }); - if (!imageContext || imageContext.type !== MessageContextType.IMAGE) { - const imageContexts = context.filter(ctx => ctx.type === MessageContextType.IMAGE); - const indexMatch = args.image_reference.match(/^\d+$/); - if (indexMatch) { - const index = parseInt(indexMatch[0]) - 1; - if (index >= 0 && index < imageContexts.length) { - const foundImage = imageContexts[index]; - if (foundImage && foundImage.type === MessageContextType.IMAGE) { - return { - image: { - mimeType: foundImage.mimeType, - data: foundImage.content, - }, - message: `Retrieved image "${foundImage.displayName}" for analysis.`, - }; - } - } - } - + if (!imageContext) { throw new Error(`No image found matching reference: ${args.image_reference}`); } return { image: { mimeType: imageContext.mimeType, data: imageContext.content, }, message: `Retrieved image "${imageContext.displayName}" for analysis.`, }; } catch (error) { throw new Error(`Cannot view image: ${error}`); } }
224-286: Extract duplicate file processing logic into a helper function.The function contains significant duplication: lines 253-266 and lines 269-282 perform nearly identical operations (MIME type extraction, filename generation, path construction, base64 decoding, file writing, and image scanning). The only difference is which image context is used.
Consider extracting the common logic:
+async function uploadImageToSandbox( + imageContext: Extract<MessageContext, { type: MessageContextType.IMAGE }>, + args: z.infer<typeof UPLOAD_IMAGE_TOOL_PARAMETERS>, + sandbox: SandboxManager, + editorEngine: EditorEngine +): Promise<string> { + const mimeType = imageContext.mimeType; + const extension = getExtensionFromMimeType(mimeType); + const filename = args.filename ? `${args.filename}.${extension}` : `${uuidv4()}.${extension}`; + const destinationPath = args.destination_path || 'public/assets/images'; + const fullPath = `${destinationPath}/${filename}`; + const base64Data = imageContext.content.replace(/^data:image\/[a-z]+;base64,/, ''); + const binaryData = base64ToUint8Array(base64Data); + await sandbox.writeBinaryFile(fullPath, binaryData); + await editorEngine.image.scanImages(); + return `Image "${imageContext.displayName}" uploaded successfully to ${fullPath}`; +} + export async function handleUploadImageTool(args: z.infer<typeof UPLOAD_IMAGE_TOOL_PARAMETERS>, editorEngine: EditorEngine): Promise<string> { try { const sandbox = editorEngine.branches.getSandboxById(args.branchId); if (!sandbox) { throw new Error(`Sandbox not found for branch ID: ${args.branchId}`); } const context = editorEngine.chat.context.context; const imageContext = context.find((ctx) => { if (ctx.type !== MessageContextType.IMAGE) { return false; } return ctx.displayName.toLowerCase().includes(args.image_reference.toLowerCase()) || args.image_reference.toLowerCase().includes(ctx.displayName.toLowerCase()); }); if (!imageContext || imageContext.type !== MessageContextType.IMAGE) { const recentImages = context.filter(ctx => ctx.type === MessageContextType.IMAGE); if (recentImages.length === 0) { throw new Error(`No image found matching reference: ${args.image_reference}`); } const mostRecentImage = recentImages[recentImages.length - 1]; if (!mostRecentImage || mostRecentImage.type === MessageContextType.IMAGE) { throw new Error(`No image found matching reference: ${args.image_reference}`); } console.warn(`No exact match for "${args.image_reference}", using most recent image: ${mostRecentImage.displayName}`); - - const mimeType = mostRecentImage.mimeType; - const extension = getExtensionFromMimeType(mimeType); - - const filename = args.filename ? `${args.filename}.${extension}` : `${uuidv4()}.${extension}`; - const destinationPath = args.destination_path || 'public/assets/images'; - const fullPath = `${destinationPath}/${filename}`; - - const base64Data = mostRecentImage.content.replace(/^data:image\/[a-z]+;base64,/, ''); - const binaryData = base64ToUint8Array(base64Data); - - await sandbox.writeBinaryFile(fullPath, binaryData); - await editorEngine.image.scanImages(); - - return `Image "${mostRecentImage.displayName}" uploaded successfully to ${fullPath}`; + return await uploadImageToSandbox(mostRecentImage, args, sandbox, editorEngine); } - const mimeType = imageContext.mimeType; - const extension = getExtensionFromMimeType(mimeType); - - const filename = args.filename ? `${args.filename}.${extension}` : `${uuidv4()}.${extension}`; - const destinationPath = args.destination_path || 'public/assets/images'; - const fullPath = `${destinationPath}/${filename}`; - - const base64Data = imageContext.content.replace(/^data:image\/[a-z]+;base64,/, ''); - const binaryData = base64ToUint8Array(base64Data); - - await sandbox.writeBinaryFile(fullPath, binaryData); - await editorEngine.image.scanImages(); - - return `Image "${imageContext.displayName}" uploaded successfully to ${fullPath}`; + return await uploadImageToSandbox(imageContext, args, sandbox, editorEngine); } catch (error) { throw new Error(`Cannot upload image: ${error}`); } }
302-309: Consider using a more concise approach.The manual byte-by-byte conversion works but could be simplified.
For better readability, consider:
function base64ToUint8Array(base64: string): Uint8Array { const binaryString = atob(base64); - const bytes = new Uint8Array(binaryString.length); - for (let i = 0; i < binaryString.length; i++) { - bytes[i] = binaryString.charCodeAt(i); - } - return bytes; + return Uint8Array.from(binaryString, char => char.charCodeAt(0)); }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx(2 hunks)apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-simple.tsx(3 hunks)apps/web/client/src/components/tools/handlers/edit.ts(2 hunks)apps/web/client/src/components/tools/tools.ts(3 hunks)packages/ai/src/prompt/provider.ts(1 hunks)packages/ai/src/tools/tools/edit.ts(1 hunks)packages/ai/src/tools/toolset.ts(2 hunks)
🧰 Additional context used
📓 Path-based instructions (6)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use the any type unless necessary
Files:
packages/ai/src/tools/toolset.tsapps/web/client/src/components/tools/tools.tsapps/web/client/src/components/tools/handlers/edit.tspackages/ai/src/prompt/provider.tsapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-simple.tsxpackages/ai/src/tools/tools/edit.ts
{apps,packages}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Avoid using the any type unless absolutely necessary
Files:
packages/ai/src/tools/toolset.tsapps/web/client/src/components/tools/tools.tsapps/web/client/src/components/tools/handlers/edit.tspackages/ai/src/prompt/provider.tsapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-simple.tsxpackages/ai/src/tools/tools/edit.ts
apps/web/client/src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
apps/web/client/src/**/*.{ts,tsx}: Use path aliases @/* and ~/* for imports that map to apps/web/client/src/*
Avoid hardcoded user-facing text; use next-intl messages/hooks insteadUse path aliases @/* and ~/* for imports mapping to src/*
Files:
apps/web/client/src/components/tools/tools.tsapps/web/client/src/components/tools/handlers/edit.tsapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-simple.tsx
apps/web/client/src/app/**/*.tsx
📄 CodeRabbit inference engine (AGENTS.md)
apps/web/client/src/app/**/*.tsx: Default to Server Components; add 'use client' when using events, state/effects, browser APIs, or client‑only libraries
Do not use process.env in client code; import env from @/env insteadAvoid hardcoded user-facing text; use next-intl messages/hooks
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-simple.tsx
apps/web/client/src/**/*.tsx
📄 CodeRabbit inference engine (AGENTS.md)
apps/web/client/src/**/*.tsx: Create MobX store instances with useState(() => new Store()) for stable references across renders
Keep the active MobX store in a useRef and perform async cleanup with setTimeout(() => storeRef.current?.clear(), 0) to avoid route-change races
Avoid useMemo for creating MobX store instances
Avoid putting the MobX store instance in effect dependency arrays if it causes loops; split concerns by domain
apps/web/client/src/**/*.tsx: Create MobX store instances with useState(() => new Store()) for stable identities across renders
Keep the active MobX store in a useRef and clean up asynchronously with setTimeout(() => storeRef.current?.clear(), 0)
Do not use useMemo to create MobX stores
Avoid placing MobX store instances in effect dependency arrays if it causes loops; split concerns instead
observer components must be client components; place a single client boundary at the feature entry; child observers need not repeat 'use client'
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-simple.tsx
apps/web/client/src/app/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Default to Server Components; add 'use client' only when using events, state/effects, browser APIs, or client-only libs
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-simple.tsx
🧬 Code graph analysis (7)
packages/ai/src/tools/toolset.ts (1)
packages/ai/src/tools/tools/edit.ts (4)
VIEW_IMAGE_TOOL_NAME(77-77)viewImageTool(85-89)UPLOAD_IMAGE_TOOL_NAME(91-91)uploadImageTool(108-112)
apps/web/client/src/components/tools/tools.ts (3)
packages/ai/src/tools/tools/edit.ts (4)
VIEW_IMAGE_TOOL_NAME(77-77)VIEW_IMAGE_TOOL_PARAMETERS(78-84)UPLOAD_IMAGE_TOOL_NAME(91-91)UPLOAD_IMAGE_TOOL_PARAMETERS(92-107)apps/web/client/src/components/store/editor/engine.ts (1)
EditorEngine(33-138)apps/web/client/src/components/tools/handlers/edit.ts (2)
handleViewImageTool(175-222)handleUploadImageTool(224-286)
apps/web/client/src/components/tools/handlers/edit.ts (3)
packages/ai/src/tools/tools/edit.ts (2)
VIEW_IMAGE_TOOL_PARAMETERS(78-84)UPLOAD_IMAGE_TOOL_PARAMETERS(92-107)apps/web/client/src/components/store/editor/engine.ts (1)
EditorEngine(33-138)apps/web/client/src/components/store/editor/chat/context.ts (2)
context(31-33)context(35-37)
packages/ai/src/prompt/provider.ts (1)
packages/ai/src/prompt/helpers.ts (1)
wrapXml(1-3)
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx (2)
packages/ai/src/tools/tools/edit.ts (4)
VIEW_IMAGE_TOOL_NAME(77-77)VIEW_IMAGE_TOOL_PARAMETERS(78-84)UPLOAD_IMAGE_TOOL_NAME(91-91)UPLOAD_IMAGE_TOOL_PARAMETERS(92-107)packages/ui/src/components/icons/index.tsx (1)
Icons(138-3667)
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-simple.tsx (1)
packages/ai/src/tools/tools/edit.ts (4)
VIEW_IMAGE_TOOL_NAME(77-77)UPLOAD_IMAGE_TOOL_NAME(91-91)VIEW_IMAGE_TOOL_PARAMETERS(78-84)UPLOAD_IMAGE_TOOL_PARAMETERS(92-107)
packages/ai/src/tools/tools/edit.ts (1)
packages/ai/src/tools/tools/branch.ts (1)
BRANCH_ID_SCHEMA(4-8)
🔇 Additional comments (7)
packages/ai/src/prompt/provider.ts (1)
115-121: LGTM! Image handling refactored to use tool-based approach.The change correctly describes available images in the prompt instead of embedding image data, allowing the AI to decide whether to view or upload them using the new tools. This aligns well with the PR objectives to avoid automatic token consumption.
packages/ai/src/tools/tools/edit.ts (2)
77-89: LGTM! VIEW_IMAGE_TOOL definition is clear and appropriate.The tool definition correctly specifies a read-only image viewing tool with appropriate parameters and a clear description.
91-112: LGTM! UPLOAD_IMAGE_TOOL definition is well-structured.The tool correctly includes:
- Required
branchIdfor write operations- Optional
destination_pathwith documented default- Optional
filenamewith documented fallback behavior- Clear description distinguishing upload from view
apps/web/client/src/components/tools/tools.ts (2)
35-38: LGTM!The imports for the new image tool constants and handlers follow the established pattern and are properly organized alongside existing tool imports.
Also applies to: 63-64
190-201: LGTM!The image tool handler registrations follow the same structure as existing tools, with proper type safety using
z.inferfor input schema validation.apps/web/client/src/components/tools/handlers/edit.ts (2)
7-12: LGTM!The new imports are properly organized and necessary for the image tool implementations.
288-300: LGTM!The MIME type to extension mapping covers common image formats and provides a sensible default fallback to 'png' for unknown types.
...ct/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
Outdated
Show resolved
Hide resolved
...ct/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
Outdated
Show resolved
Hide resolved
...ect/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-simple.tsx
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (3)
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx (3)
77-77: Consider extracting shared container styling.Both ViewImageTool and UploadImageTool blocks use identical container styling (
flex flex-col gap-2 p-3 border rounded-lg bg-background-secondary). Extract this into a shared component or constant to reduce duplication.Also applies to: 100-100
73-94: Use next-intl for user-facing text.The ViewImageTool rendering block contains hardcoded user-facing strings that violate the i18n guidelines:
- Line 80: "View Image"
- Line 84: "Image:"
As per coding guidelines.
Based on coding guidelines
96-127: Use next-intl for user-facing text.The UploadImageTool rendering block contains hardcoded user-facing strings that violate the i18n guidelines:
- Line 103: "Upload Image"
- Line 107: "Image:"
- Line 112: "Destination:"
- Line 117: "Filename:"
As per coding guidelines.
Based on coding guidelines
🧹 Nitpick comments (7)
packages/ai/src/agents/tool-lookup.ts (1)
1-1: Consider splitting the long import statement.The import statement exceeds 120 characters. While functional, splitting it into multiple lines would improve readability.
Apply this diff to split the import:
-import { ListFilesTool, ReadFileTool, BashReadTool, OnlookInstructionsTool, ReadStyleGuideTool, ListBranchesTool, ScrapeUrlTool, WebSearchTool, GlobTool, GrepTool, TypecheckTool, CheckErrorsTool, SearchReplaceEditTool, SearchReplaceMultiEditFileTool, FuzzyEditFileTool, WriteFileTool, BashEditTool, SandboxTool, TerminalCommandTool, ViewImageTool, UploadImageTool } from "../tools"; +import { + ListFilesTool, + ReadFileTool, + BashReadTool, + OnlookInstructionsTool, + ReadStyleGuideTool, + ListBranchesTool, + ScrapeUrlTool, + WebSearchTool, + GlobTool, + GrepTool, + TypecheckTool, + CheckErrorsTool, + SearchReplaceEditTool, + SearchReplaceMultiEditFileTool, + FuzzyEditFileTool, + WriteFileTool, + BashEditTool, + SandboxTool, + TerminalCommandTool, + ViewImageTool, + UploadImageTool, +} from "../tools";packages/ai/src/tools/classes/view-image.ts (2)
25-33: Refactor inefficient image lookup logic.Line 32 filters the context array on every iteration of the
find()call, which is inefficient. The condition also makes the control flow confusing since the numeric index handling is duplicated in the fallback logic (lines 36-52).Simplify by checking name matches first, then handling numeric indices separately:
const context = editorEngine.chat.context.context; + const ref = args.image_reference.toLowerCase(); + + // First try matching by name const imageContext = context.find((ctx) => { if (ctx.type !== MessageContextType.IMAGE) { return false; } - const ref = args.image_reference.toLowerCase(); - return ctx.displayName.toLowerCase().includes(ref) || - ref.includes(ctx.displayName.toLowerCase()) || - ref.match(/^\d+$/) && context.filter(c => c.type === MessageContextType.IMAGE)[parseInt(ref) - 1] === ctx; + return ctx.displayName.toLowerCase().includes(ref) || + ref.includes(ctx.displayName.toLowerCase()); });The numeric index handling in the fallback (lines 36-52) already covers that case correctly.
69-74: Add ellipsis to indicate truncation.Line 71 truncates the image reference to 20 characters without indicating truncation. This could confuse users when viewing long filenames.
Apply this diff to add an ellipsis:
getLabel(input?: z.infer<typeof ViewImageTool.parameters>): string { if (input?.image_reference) { - return 'Viewing image ' + input.image_reference.substring(0, 20); + const ref = input.image_reference; + return 'Viewing image ' + (ref.length > 20 ? ref.substring(0, 20) + '...' : ref); } return 'Viewing image'; }packages/ai/src/tools/classes/upload-image.ts (4)
56-58: Remove redundant type check.Line 56 checks if
mostRecentImage.type !== MessageContextType.IMAGEafter already filtering for IMAGE types on line 50. This check is redundant unless TypeScript's type narrowing doesn't work properly here.- const mostRecentImage = recentImages[recentImages.length - 1]; - if (!mostRecentImage || mostRecentImage.type !== MessageContextType.IMAGE) { - throw new Error(`No image found matching reference: ${args.image_reference}`); - } + const mostRecentImage = recentImages[recentImages.length - 1]; + if (!mostRecentImage) { + throw new Error(`No image found matching reference: ${args.image_reference}`); + }
92-94: Improve error handling to preserve error context.The current error wrapping at line 93 converts all errors to strings, which loses type information and could expose internal stack traces to users. Consider preserving the original error or providing more specific error messages.
} catch (error) { - throw new Error(`Cannot upload image: ${error}`); + const message = error instanceof Error ? error.message : String(error); + throw new Error(`Cannot upload image: ${message}`); }
104-116: Consider adding support for modern image formats.The MIME type mapping covers common formats but is missing some modern ones like AVIF, HEIC/HEIF, and ICO. The fallback to 'png' is reasonable but could be problematic for formats that can't be safely renamed.
Consider extending the mapping:
private getExtensionFromMimeType(mimeType: string): string { const mimeToExt: Record<string, string> = { 'image/jpeg': 'jpg', 'image/jpg': 'jpg', 'image/png': 'png', 'image/gif': 'gif', 'image/webp': 'webp', 'image/svg+xml': 'svg', 'image/bmp': 'bmp', 'image/tiff': 'tiff', + 'image/avif': 'avif', + 'image/heic': 'heic', + 'image/heif': 'heif', + 'image/x-icon': 'ico', + 'image/vnd.microsoft.icon': 'ico', }; return mimeToExt[mimeType.toLowerCase()] || 'png'; }
72-73: Avoid full directory scan on each upload
editorEngine.image.scanImages()iterates over the entireactiveSandbox.filesarray (O(n) per call), so invoking it after everywriteBinaryFilemay degrade upload throughput. Consider updating_imagePathsincrementally (e.g. append the new path) or batching/debouncing the scanImages calls instead of performing a full filter each time.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx(2 hunks)apps/web/client/src/components/tools/tools.ts(1 hunks)packages/ai/src/agents/tool-lookup.ts(4 hunks)packages/ai/src/prompt/provider.ts(1 hunks)packages/ai/src/tools/classes/index.ts(1 hunks)packages/ai/src/tools/classes/upload-image.ts(1 hunks)packages/ai/src/tools/classes/view-image.ts(1 hunks)packages/ai/src/tools/toolset.ts(1 hunks)
✅ Files skipped from review due to trivial changes (2)
- packages/ai/src/tools/toolset.ts
- apps/web/client/src/components/tools/tools.ts
🧰 Additional context used
📓 Path-based instructions (6)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use the any type unless necessary
Files:
packages/ai/src/agents/tool-lookup.tspackages/ai/src/tools/classes/index.tsapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxpackages/ai/src/tools/classes/view-image.tspackages/ai/src/tools/classes/upload-image.tspackages/ai/src/prompt/provider.ts
{apps,packages}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Avoid using the any type unless absolutely necessary
Files:
packages/ai/src/agents/tool-lookup.tspackages/ai/src/tools/classes/index.tsapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxpackages/ai/src/tools/classes/view-image.tspackages/ai/src/tools/classes/upload-image.tspackages/ai/src/prompt/provider.ts
apps/web/client/src/app/**/*.tsx
📄 CodeRabbit inference engine (AGENTS.md)
apps/web/client/src/app/**/*.tsx: Default to Server Components; add 'use client' when using events, state/effects, browser APIs, or client‑only libraries
Do not use process.env in client code; import env from @/env insteadAvoid hardcoded user-facing text; use next-intl messages/hooks
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
apps/web/client/src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
apps/web/client/src/**/*.{ts,tsx}: Use path aliases @/* and ~/* for imports that map to apps/web/client/src/*
Avoid hardcoded user-facing text; use next-intl messages/hooks insteadUse path aliases @/* and ~/* for imports mapping to src/*
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
apps/web/client/src/**/*.tsx
📄 CodeRabbit inference engine (AGENTS.md)
apps/web/client/src/**/*.tsx: Create MobX store instances with useState(() => new Store()) for stable references across renders
Keep the active MobX store in a useRef and perform async cleanup with setTimeout(() => storeRef.current?.clear(), 0) to avoid route-change races
Avoid useMemo for creating MobX store instances
Avoid putting the MobX store instance in effect dependency arrays if it causes loops; split concerns by domain
apps/web/client/src/**/*.tsx: Create MobX store instances with useState(() => new Store()) for stable identities across renders
Keep the active MobX store in a useRef and clean up asynchronously with setTimeout(() => storeRef.current?.clear(), 0)
Do not use useMemo to create MobX stores
Avoid placing MobX store instances in effect dependency arrays if it causes loops; split concerns instead
observer components must be client components; place a single client boundary at the feature entry; child observers need not repeat 'use client'
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
apps/web/client/src/app/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Default to Server Components; add 'use client' only when using events, state/effects, browser APIs, or client-only libs
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
🧠 Learnings (2)
📚 Learning: 2025-09-14T01:44:21.209Z
Learnt from: CR
PR: onlook-dev/onlook#0
File: AGENTS.md:0-0
Timestamp: 2025-09-14T01:44:21.209Z
Learning: Applies to apps/web/client/src/**/*.{ts,tsx} : Avoid hardcoded user-facing text; use next-intl messages/hooks instead
Applied to files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
📚 Learning: 2025-09-16T19:22:52.461Z
Learnt from: CR
PR: onlook-dev/onlook#0
File: CLAUDE.md:0-0
Timestamp: 2025-09-16T19:22:52.461Z
Learning: Applies to apps/web/client/src/app/**/*.tsx : Avoid hardcoded user-facing text; use next-intl messages/hooks
Applied to files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
🧬 Code graph analysis (3)
packages/ai/src/agents/tool-lookup.ts (1)
packages/ai/src/tools/classes/index.ts (2)
ViewImageTool(19-19)UploadImageTool(18-18)
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx (1)
packages/ai/src/tools/classes/index.ts (2)
ViewImageTool(19-19)UploadImageTool(18-18)
packages/ai/src/prompt/provider.ts (1)
packages/ai/src/prompt/helpers.ts (1)
wrapXml(1-3)
🔇 Additional comments (8)
packages/ai/src/prompt/provider.ts (1)
115-123: LGTM! Image guidance properly integrated.The new image-handling directive correctly lists available images and explains the view_image and upload_image tools to the AI. The XML wrapping is consistent with the rest of the prompt construction.
packages/ai/src/tools/classes/index.ts (1)
18-19: LGTM! New tools properly exported.The new image tools follow the existing export pattern and are correctly positioned alphabetically.
packages/ai/src/agents/tool-lookup.ts (1)
24-25: LGTM! Tools correctly categorized and registered.ViewImageTool is appropriately placed in readOnlyRootTools (since viewing doesn't modify state), and UploadImageTool is correctly in editOnlyRootTools (since uploading writes to the filesystem).
Also applies to: 43-43, 54-54
packages/ai/src/tools/classes/upload-image.ts (5)
1-7: LGTM!Imports are correctly structured and compatible with the library versions in use (uuid v11.1.0, zod v4.1.3).
9-28: LGTM!Tool metadata is well-defined with clear descriptions. The Zod schema correctly uses v4-compatible patterns.
40-47: Review the bidirectional substring matching logic.The current matching logic uses bidirectional case-insensitive substring matching, which could match multiple images unexpectedly. For example, if a user references "logo", it would match both "logo.png" and "company-logo.jpg". Consider whether this is the intended behavior or if exact matching or a more specific matching strategy would be better.
Please verify this matching behavior aligns with user expectations, particularly when multiple similarly named images exist in the chat context.
97-102: LGTM!The label generation logic is clear and provides appropriate user feedback.
118-125: LGTM!The base64 to Uint8Array conversion is correctly implemented and returns the proper type (Uint8Array) compatible with uuid v11 and modern APIs.
| } | ||
|
|
||
| private async uploadImageToSandbox( | ||
| imageContext: Extract<import('@onlook/models').MessageContext, { type: MessageContextType.IMAGE }>, |
There was a problem hiding this comment.
nit: should not use inline import for type
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| const ref = args.image_reference.toLowerCase(); | ||
| return ctx.displayName.toLowerCase().includes(ref) || | ||
| ref.includes(ctx.displayName.toLowerCase()) || | ||
| ref.match(/^\d+$/) && context.filter(c => c.type === MessageContextType.IMAGE)[parseInt(ref) - 1] === ctx; |
There was a problem hiding this comment.
There's an issue with the numeric index matching logic in the image reference lookup. When checking if a reference is a numeric index (ref.match(/^\d+$/)), the code attempts to find the image at that position using the entire context array rather than just the image contexts.
This creates a potential mismatch since the index is meant to reference the nth image, not the nth context item. The code should use the filtered imageContexts array (which is created later in the error handling section) for this lookup instead:
// Current problematic code
ref.match(/^\d+$/) && context.filter(c => c.type === MessageContextType.IMAGE)[parseInt(ref) - 1] === ctx;
// Suggested approach
// First filter for image contexts, then check if the current context is at the specified index
const imageContexts = context.filter(c => c.type === MessageContextType.IMAGE);
ref.match(/^\d+$/) && imageContexts[parseInt(ref) - 1] === ctx;This would ensure that numeric references correctly match the intended image by position in the image list.
| const ref = args.image_reference.toLowerCase(); | |
| return ctx.displayName.toLowerCase().includes(ref) || | |
| ref.includes(ctx.displayName.toLowerCase()) || | |
| ref.match(/^\d+$/) && context.filter(c => c.type === MessageContextType.IMAGE)[parseInt(ref) - 1] === ctx; | |
| const ref = args.image_reference.toLowerCase(); | |
| const imageContexts = context.filter(c => c.type === MessageContextType.IMAGE); | |
| return ctx.displayName.toLowerCase().includes(ref) || | |
| ref.includes(ctx.displayName.toLowerCase()) || | |
| ref.match(/^\d+$/) && imageContexts[parseInt(ref) - 1] === ctx; |
Spotted by Diamond
Is this helpful? React 👍 or 👎 to let us know.
| } | ||
|
|
||
| const context = editorEngine.chat.context.context; | ||
| const imageContext = context.find((ctx) => { |
There was a problem hiding this comment.
We should probably use an actual ID for reference instead of the name by updating the context type. What do you think? And should we store the image in persistent storage instead of using context which is ephemeral so that the reference can be kept if the page is refreshed, etc?
| return fullPath; | ||
| } | ||
|
|
||
| private getExtensionFromMimeType(mimeType: string): string { |
There was a problem hiding this comment.
can we use the existing mime-lite lib for this?
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (3)
packages/ai/src/tools/classes/view-image.ts (1)
25-27: Previous numeric index issue has been resolved.The past review comment flagged an issue with numeric index matching logic. The current implementation correctly uses exact ID matching (
ctx.id === args.image_id) instead of any numeric index or pattern matching, which resolves that concern.apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx (2)
77-125: Consider extracting shared tool display styling.Both
ViewImageToolandUploadImageToolrendering blocks share identical container styling and similar structural patterns. Consider extracting a shared component (e.g.,ToolResultCard) to reduce duplication and improve maintainability.
73-127: Use next-intl for user-facing text.Multiple hardcoded English strings violate the i18n coding guidelines. The following strings should be extracted to translation keys:
ViewImageTool block:
- Line 80: "View Image"
- Line 84: "Image ID:"
UploadImageTool block:
- Line 103: "Upload Image"
- Line 107: "Image ID:"
- Line 112: "Destination:"
- Line 117: "Filename:"
Consider importing
useTranslationsand replacing these with translation keys liket('chat.tool.viewImage'),t('chat.tool.imageId'), etc.As per coding guidelines.
🧹 Nitpick comments (1)
packages/ai/src/tools/classes/upload-image.ts (1)
66-80: Prefer explicit type overanyfor sandbox parameter.Line 69 uses
anyfor thesandboxparameter. Consider importing and using the proper type (likelySandboxManagerbased on theEditorEnginecontext) to improve type safety and catch potential errors at compile time.As per coding guidelines: Avoid using the
anytype unless absolutely necessary.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
bun.lockis excluded by!**/*.lock
📒 Files selected for processing (9)
apps/web/client/src/app/_components/hero/create.tsx(2 hunks)apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-input/index.tsx(3 hunks)apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx(2 hunks)apps/web/client/src/app/project/[id]/_hooks/use-start-project.tsx(2 hunks)packages/ai/package.json(1 hunks)packages/ai/src/prompt/provider.ts(1 hunks)packages/ai/src/tools/classes/upload-image.ts(1 hunks)packages/ai/src/tools/classes/view-image.ts(1 hunks)packages/models/src/chat/message/context.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/ai/src/prompt/provider.ts
🧰 Additional context used
📓 Path-based instructions (6)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Do not use the any type unless necessary
Files:
packages/models/src/chat/message/context.tsapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_hooks/use-start-project.tsxapps/web/client/src/app/_components/hero/create.tsxpackages/ai/src/tools/classes/upload-image.tsapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-input/index.tsxpackages/ai/src/tools/classes/view-image.ts
{apps,packages}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Avoid using the any type unless absolutely necessary
Files:
packages/models/src/chat/message/context.tsapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_hooks/use-start-project.tsxapps/web/client/src/app/_components/hero/create.tsxpackages/ai/src/tools/classes/upload-image.tsapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-input/index.tsxpackages/ai/src/tools/classes/view-image.ts
apps/web/client/src/app/**/*.tsx
📄 CodeRabbit inference engine (AGENTS.md)
apps/web/client/src/app/**/*.tsx: Default to Server Components; add 'use client' when using events, state/effects, browser APIs, or client‑only libraries
Do not use process.env in client code; import env from @/env insteadAvoid hardcoded user-facing text; use next-intl messages/hooks
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_hooks/use-start-project.tsxapps/web/client/src/app/_components/hero/create.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-input/index.tsx
apps/web/client/src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
apps/web/client/src/**/*.{ts,tsx}: Use path aliases @/* and ~/* for imports that map to apps/web/client/src/*
Avoid hardcoded user-facing text; use next-intl messages/hooks insteadUse path aliases @/* and ~/* for imports mapping to src/*
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_hooks/use-start-project.tsxapps/web/client/src/app/_components/hero/create.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-input/index.tsx
apps/web/client/src/**/*.tsx
📄 CodeRabbit inference engine (AGENTS.md)
apps/web/client/src/**/*.tsx: Create MobX store instances with useState(() => new Store()) for stable references across renders
Keep the active MobX store in a useRef and perform async cleanup with setTimeout(() => storeRef.current?.clear(), 0) to avoid route-change races
Avoid useMemo for creating MobX store instances
Avoid putting the MobX store instance in effect dependency arrays if it causes loops; split concerns by domain
apps/web/client/src/**/*.tsx: Create MobX store instances with useState(() => new Store()) for stable identities across renders
Keep the active MobX store in a useRef and clean up asynchronously with setTimeout(() => storeRef.current?.clear(), 0)
Do not use useMemo to create MobX stores
Avoid placing MobX store instances in effect dependency arrays if it causes loops; split concerns instead
observer components must be client components; place a single client boundary at the feature entry; child observers need not repeat 'use client'
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_hooks/use-start-project.tsxapps/web/client/src/app/_components/hero/create.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-input/index.tsx
apps/web/client/src/app/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Default to Server Components; add 'use client' only when using events, state/effects, browser APIs, or client-only libs
Files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsxapps/web/client/src/app/project/[id]/_hooks/use-start-project.tsxapps/web/client/src/app/_components/hero/create.tsxapps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-input/index.tsx
🧠 Learnings (2)
📚 Learning: 2025-09-14T01:44:21.209Z
Learnt from: CR
PR: onlook-dev/onlook#0
File: AGENTS.md:0-0
Timestamp: 2025-09-14T01:44:21.209Z
Learning: Applies to apps/web/client/src/**/*.{ts,tsx} : Avoid hardcoded user-facing text; use next-intl messages/hooks instead
Applied to files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
📚 Learning: 2025-09-16T19:22:52.461Z
Learnt from: CR
PR: onlook-dev/onlook#0
File: CLAUDE.md:0-0
Timestamp: 2025-09-16T19:22:52.461Z
Learning: Applies to apps/web/client/src/app/**/*.tsx : Avoid hardcoded user-facing text; use next-intl messages/hooks
Applied to files:
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx
🧬 Code graph analysis (3)
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx (3)
packages/ai/src/tools/classes/view-image.ts (1)
ViewImageTool(7-51)packages/ui/src/components/icons/index.tsx (1)
Icons(138-3667)packages/ai/src/tools/classes/upload-image.ts (1)
UploadImageTool(10-90)
packages/ai/src/tools/classes/upload-image.ts (3)
packages/ai/src/tools/shared/type.ts (1)
BRANCH_ID_SCHEMA(3-7)apps/web/client/src/components/store/editor/engine.ts (1)
EditorEngine(34-140)packages/models/src/chat/message/context.ts (1)
MessageContext(54-60)
packages/ai/src/tools/classes/view-image.ts (2)
packages/ui/src/components/icons/index.tsx (1)
Icons(138-3667)apps/web/client/src/components/store/editor/engine.ts (1)
EditorEngine(34-140)
🔇 Additional comments (18)
packages/ai/package.json (1)
49-49: LGTM! Appropriate dependency for MIME type handling.The
mime-litedependency is correctly added to support file extension detection in the newUploadImageTool. Version 1.0.3 is the latest stable release and provides the minimal MIME type mapping needed for this feature.Based on learnings.
packages/models/src/chat/message/context.ts (1)
41-41: LGTM! Essential addition for image identification.The
idfield enables unique identification of image contexts, which is required by the newViewImageToolandUploadImageTool. The related changes show that all image context creation points have been updated to include UUID-based IDs.packages/ai/src/tools/classes/view-image.ts (2)
1-6: LGTM! Clean and appropriate imports.All imports are correctly organized and necessary for the tool implementation.
7-50: LGTM! Well-structured tool implementation.The
ViewImageToolclass is correctly implemented with:
- Appropriate static metadata for tool registration
- Clear Zod schema validation for parameters
- Proper error handling with descriptive messages
- Type-safe image context lookup and validation
- User-friendly label generation
The exact ID matching approach eliminates ambiguity and is more reliable than name-based or index-based lookups.
apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-input/index.tsx (3)
19-19: LGTM! Appropriate import for unique ID generation.The
uuidimport is correctly added to generate unique identifiers for image contexts.
212-218: LGTM! Proper ID assignment for image contexts.The UUID is correctly generated and assigned when handling pasted or dropped images, ensuring each image context has a unique identifier required by the updated
ImageMessageContexttype.
265-271: LGTM! Consistent ID generation for screenshots.The UUID assignment for screenshot image contexts follows the same pattern as regular image handling, ensuring consistency across all image context creation points.
apps/web/client/src/app/project/[id]/_hooks/use-start-project.tsx (2)
16-16: LGTM! Appropriate import for image context IDs.The
uuidimport is correctly added to generate unique identifiers when resuming project creation with images.
91-99: LGTM! Proper image context transformation.The code correctly transforms creation request images into message context images with unique IDs, maintaining consistency with the updated
ImageMessageContexttype.apps/web/client/src/app/_components/hero/create.tsx (2)
19-19: LGTM! Appropriate import for image ID generation.The
uuidimport is correctly added to support unique identification of image contexts in the project creation flow.
199-205: LGTM! Complete image context creation.The
createImageMessageContexthelper correctly constructs anImageMessageContextwith all required fields, including the newidfield with a unique UUID.apps/web/client/src/app/project/[id]/_components/right-panel/chat-tab/chat-messages/message-content/tool-call-display.tsx (1)
1-3: LGTM! Imports correctly updated for new tools.The imports properly include the new
ViewImageToolandUploadImageToolclasses along with theIconscomponent needed for rendering.packages/ai/src/tools/classes/upload-image.ts (6)
1-8: LGTM! Imports are correctly organized.All necessary dependencies are imported appropriately. The
mime-litelibrary is correctly imported for runtime MIME type handling.
10-29: LGTM! Well-defined tool metadata.The static properties are correctly structured with:
- Clear tool name and description
- Comprehensive parameter schema with appropriate optional fields
- Good default value documentation in descriptions
31-57: LGTM! Robust error handling and clear flow.The
handlemethod correctly:
- Validates sandbox existence before proceeding
- Uses exact ID matching for image lookup
- Triggers image indexing after upload
- Provides descriptive error messages
66-80: LGTM! Well-implemented upload logic.The
uploadImageToSandboxmethod correctly:
- Uses
mime-litefor extension detection with a sensible fallback- Handles both custom filenames and UUID generation
- Properly strips the base64 data URI prefix with a comprehensive regex pattern
- Converts base64 to binary format for file writing
Based on learnings (mime-lite usage).
82-89: LGTM! Correct base64 to binary conversion.The
base64ToUint8Arraymethod uses the standard browser-compatible approach withatobto decode base64 strings and manually construct the byte array, which is appropriate for this context.
59-64: LGTM! User-friendly label generation.The
getLabelmethod provides appropriate truncated labels for tool display, maintaining consistency withViewImageTool.
|
@Kitenite Also, the upload image tool works by uploading the image to the container/sandbox, and the reference to the image is kept in chat. It can be used in follow-up chat prompts. It is not meant to be kept in the chat context, as it was meant to be uploaded. About persistent storage, we can look into that in a separate PR, users might be logged in multiple devices, so storing it locally might not be a long-term solution. |
Description
Added two new AI tools (
view_imageandupload_image) that enable intelligent handling of images attached to chat messages. Instead of automatically sending all images to the AI (which consumes tokens), images are now stored separately, and the AI determines the appropriate action based on user intent.Changes
view_image: Retrieves image data for use by AI chat contextupload_image: Saves images to project file system (default:public/assets/images/)Related Issues
Type of Change
Testing
Screenshots (if applicable)
Additional Notes
Important
Adds
view_imageandupload_imagetools for AI-driven image handling in chat, with UI and backend support.view_image: Retrieves image data for AI analysis without saving.upload_image: Saves images topublic/assets/images/with optional custom filename.handleViewImageToolinedit.ts: Retrieves image by reference from chat context.handleUploadImageToolinedit.ts: Uploads image to project file system, handles missing references by using the most recent image.ToolCallDisplayandToolCallSimpleupdated to supportview_imageandupload_imagetools.getHydratedUserMessageinprovider.ts: Adds image references to prompt for AI to decide action.This description was created by
for c5daeaf. You can customize this summary. It will automatically update as commits are pushed.
Summary by CodeRabbit
New Features
Improvements
Other