Add gallery component for displaying multiple media items#2129
Add gallery component for displaying multiple media items#2129
Conversation
Add a new built-in tool that the LLM can call within the MCP flow to present images, videos, and audio in a horizontal scrollable gallery. The gallery renders prominently (not hidden in the tool accordion) and the model is informed via the system prompt when to use it. - New builtinTools.ts module with display_gallery schema and local execution - ToolGallery.svelte with snap-scroll, arrow nav, lightbox, video/audio support - Built-in tools merged into MCP flow alongside external MCP server tools - toolInvocation intercepts built-in tools before MCP dispatch - Gallery-specific guidance added to tool system prompt https://claude.ai/code/session_019mXtHGmqM5d1zijZ3mVZB6
Replace the built-in display_gallery tool with a simpler <gallery> markup approach that follows the same pattern as <think> blocks. The model outputs <gallery title="...">JSON</gallery> inline in its response text, and the frontend detects, parses, and renders it as a horizontal scrollable gallery. This approach is better because: - No server-side MCP pipeline changes needed - Works even when tool-calling is disabled - Gallery can appear inline with surrounding text naturally - Follows the existing <think> block pattern Changes: - Revert builtinTools.ts, runMcpFlow.ts, toolInvocation.ts, ToolUpdate.svelte - ChatMessage.svelte: add gallery regex detection, JSON parser, renderTextContent snippet that splits text by <gallery> blocks (works with <think> too) - toolPrompt.ts: describe <gallery> markup format in system prompt - ToolGallery.svelte component is unchanged (now used from ChatMessage) https://claude.ai/code/session_019mXtHGmqM5d1zijZ3mVZB6
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e3f130abe1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| {#if item.media_type === "image"} | ||
| <button | ||
| class="block w-full cursor-zoom-in overflow-hidden rounded-lg border border-gray-200 dark:border-gray-700" | ||
| onclick={() => (lightboxSrc = item.url)} |
There was a problem hiding this comment.
Clicking a gallery image fires this handler and also bubbles to ChatMessage.handleContentClick, which opens a second ImageLightbox for any IMG target. In assistant messages that creates stacked modals (often requiring multiple closes and conflicting body-scroll state), so gallery image interactions are currently broken unless the click event is stopped here or lightbox ownership is centralized.
Useful? React with 👍 / 👎.
2 participants
Summary
This PR introduces a new
ToolGallerycomponent that allows displaying multiple media items (images, videos, and audio) in a horizontal scrollable gallery format. The component is integrated into the chat message rendering system to support a new<gallery>block syntax.Key Changes
New
ToolGallery.sveltecomponent: A reusable gallery component that displays media items in a horizontal scrollable container with:Updated
ChatMessage.svelte:<gallery>...</gallery>parseGalleryBlock()function to extract title and JSON-formatted media items from gallery blocksrenderTextContent()snippet to handle rendering of both markdown content and gallery blocksToolGallerycomponent into message rendering pipelineUpdated
toolPrompt.ts:Implementation Details
urlandmedia_typefields;title,description, andthumbnail_urlare optional<gallery title="...">JSON</gallery>https://claude.ai/code/session_019mXtHGmqM5d1zijZ3mVZB6