Merge pull request #13 from achetronic/feat/add-message-splitting

Feat/add message splitting

Author: achetronic

.agents/CLIENT_DESIGN.md

@@ -181,6 +181,9 @@ server/clients/
 ├── provider.go          — Provider interface, Schema type alias
 ├── registry.go          — Global registry: Register(), ValidateConfig() with oneOf
 ├── executor.go          — Shared execution logic (webhook + cron)
+├── msgutil/
+│   ├── msgutil.go       — Shared message validation and splitting utilities
+│   └── msgutil_test.go  — Tests for validation and splitting
 ├── direct/spec.go       — Direct provider (empty schema)
 ├── telegram/
 │   ├── spec.go          — Telegram provider (JSON Schema with x-format, enum)
@@ -444,8 +447,77 @@ If the message has a caption, it goes as the text part. If no caption, the text
 
 ### File size validation
 
-20MB limit enforced before downloading. Telegram bot API limits files to 20MB anyway. For Slack, check `file.Size` before calling `downloadSlackFile()`.
+5MB per file, 10MB total per message, max 10 files. Enforced client-side before downloading. Telegram bot API limits files to 20MB anyway. For Slack, check `file.Size` before calling `downloadSlackFile()`.
 
 ### A2A (future)
 
 A2A agent cards (`server/a2a/handler.go`) currently declare `DefaultInputModes: []string{"text/plain"}`. When A2A file support is added, include additional MIME types (`image/*`, `application/pdf`, etc.) and convert A2A `FilePart` → `genai.Part{InlineData}` in the executor.
+
+## Message Size Handling (Implemented)
+
+### Package: `server/clients/msgutil/`
+
+Shared utility package for message validation and splitting. Both Telegram and Slack clients import it. Clients stay decoupled — each calls the utility with its own platform limits.
+
+### Constants
+
+| Constant | Value | Usage |
+|----------|-------|-------|
+| `TelegramMaxMessageLength` | 4096 | Telegram API limit per message |
+| `SlackMaxMessageLength` | 39000 | Slack API limit per message block |
+| `DefaultMaxInputLength` | 16000 | Max inbound user message length |
+
+### Functions
+
+**`ValidateInputLength(text string, maxLen int) (string, bool)`**
+- Truncates at `maxLen` runes (unicode-safe), appends `\n\n[message truncated]`
+- Returns the (possibly truncated) text and whether truncation occurred
+- Applied at client entry points before calling the agent
+
+**`SplitMessage(text string, maxLen int) []string`**
+- Splits into chunks respecting `maxLen` runes per chunk
+- Split priority: paragraph (`\n\n`) > line (`\n`) > word (space) > hard cut
+- Returns `[]string` — all chunks non-empty, within limit
+
+### Where it's applied
+
+| Client | Inbound | Outbound |
+|--------|---------|----------|
+| **Telegram** | `handleMessage()` validates `msg.Text`; `handleVoice()` validates transcribed text | `sendResponse()` splits via `SplitMessage(text, 4096)`, sends chunks sequentially |
+| **Slack** | `processMessage()` validates text (covers DMs + audio clips) | `postMessage()` splits via `SplitMessage(text, 39000)`, posts chunks sequentially |
+| **Voice UI** | No validation (browser input is bounded) | No splitting (browser has no render limit) |
+| **Executor** | No validation (prompts are from commands/webhooks, admin-controlled) | No splitting (returns string to HTTP caller) |
+
+---
+
+## Artifact Delivery
+
+When an LLM uses `save_artifact` during a `/run` call, clients automatically deliver the new artifact as a file attachment. The flow:
+
+1. **Before `/run`**: Client calls `GET /apps/{agent}/users/{user}/sessions/{session}/artifacts` to snapshot existing artifact names
+2. **After `/run`**: Client calls the same endpoint again and diffs the two lists
+3. **New artifacts**: Each new name is downloaded via `GET .../artifacts/{name}` and sent as a file
+
+### Delivery per client
+
+| Client | Method | Details |
+|--------|--------|---------|
+| **Telegram** | `ctx.Bot().SendDocument()` with `tu.FileFromReader()` | Artifact name used as filename |
+| **Slack** | `c.api.UploadFileV2()` | Artifact name as filename + title, respects thread |
+| **Voice UI** | Not yet implemented | Would need download button in UI |
+
+### Artifact REST response format
+
+The ADK artifact endpoint returns a `genai.Part` JSON:
+- **Text artifacts**: `{"text": "content..."}`
+- **Binary artifacts**: `{"inlineData": {"mimeType": "...", "data": "<base64>"}}`
+
+### Key files
+
+| File | Role |
+|------|------|
+| `server/agent/tools/artifacts/toolset.go` | Toolset with save/load/list tools |
+| `server/agent/base_toolset.go` | Wires artifact toolset into all agents |
+| `server/agent/agent.go` | Creates `artifactfs.NewFilesystemService()`, sets `launcherCfg.ArtifactService` |
+| `server/clients/telegram/bot.go` | `listArtifacts()`, `downloadArtifact()`, `sendNewArtifacts()` |
+| `server/clients/slack/bot.go` | Same three methods, adapted for Slack API |

.agents/DECISIONS.md

@@ -361,3 +361,69 @@ the existing value.
 and recreate. Non-secret entities remain intact.
 
 **Do not**: Return secret values in GET responses. Do not store secrets in config.yaml.
+
+---
+
+## Message size validation and splitting via shared msgutil package
+
+**Date**: 2026-02-23
+**Status**: Implemented
+
+Large inbound messages and oversized outbound responses are handled by a shared utility package `server/clients/msgutil/`. Both Telegram and Slack clients import it — the logic is DRY and testable, while clients remain decoupled from each other and from the ADK API.
+
+**Inbound validation**: `ValidateInputLength(text, maxLen)` truncates messages exceeding 16K runes (unicode-safe) and appends `[message truncated]`. Applied in both clients before calling the agent.
+
+**Outbound splitting**: `SplitMessage(text, maxLen)` breaks responses into platform-safe chunks. Split priority: paragraph boundaries (`\n\n`) > line boundaries (`\n`) > word boundaries (space) > hard cut. Telegram uses 4096, Slack uses 39000.
+
+**Platform constants**:
+- `TelegramMaxMessageLength = 4096`
+- `SlackMaxMessageLength = 39000`
+- `DefaultMaxInputLength = 16000`
+
+**Where validation happens**:
+- Telegram: `handleMessage()` validates `msg.Text`, `handleVoice()` validates transcribed text — both before `callAgent()`
+- Slack: `processMessage()` validates `text` before building the request — covers both DMs and audio clips (which flow through `processMessage`)
+- Voice UI: no splitting needed (browser has no render limit)
+- Executor: no splitting needed (returns string to HTTP caller)
+
+**Do not**: Validate or split inside `callAgent()` / the ADK request path. Keep it at the client entry/exit points so each client controls its own limits. Do not add platform-specific logic to the shared package — it only provides generic split/validate functions with configurable limits.
+
+---
+
+## 17. Artifact Toolset — Universal via Base Toolset, No Delete
+
+**Date**: 2025-02-23
+
+All agents get the artifact toolset (save/load/list) unconditionally via `base_toolset.go`, not opt-in per agent. This avoids config complexity and ensures every agent can produce files for users.
+
+**No delete tool**: ADK's `agent.Artifacts` interface (exposed via `tool.Context`) has Save, Load, List, and LoadVersion — but no Delete. Delete exists only on `artifact.Service` directly. Rather than breaking the abstraction by passing the raw service into tools, we omit delete. Artifacts are versioned and session-scoped, so stale artifacts are naturally cleaned up when sessions expire.
+
+**Storage**: `adk-utils-go/artifact/filesystem` — filesystem-backed `artifact.Service` implementation. Stores artifacts as JSON at `data/artifacts/{appName}/{userID}/{sessionID}/{fileName}/{version}.json`. Supports versioning and user-scoped artifacts. Data persists across restarts.
+
+**Client delivery**: Telegram and Slack clients list artifacts before and after each `/run` call, diff the lists, and deliver new artifacts as file attachments (Telegram: `SendDocument`, Slack: `UploadFileV2`). Artifacts are always files, never inlined in chat text.
+
+**Files**: `server/agent/tools/artifacts/toolset.go` (toolset), `server/agent/base_toolset.go` (wiring), `server/agent/agent.go` (FilesystemService + launcher config), `server/clients/telegram/bot.go` and `server/clients/slack/bot.go` (delivery).
+
+---
+
+## 18. Multimodal Adapter Parity — Error on Unsupported Types
+
+**Date**: 2026-02-23
+
+When an adapter receives `genai.Part{InlineData}` with a MIME type it can't translate, it returns an error — **not** `nil` (silent drop). This matches Gemini's native behavior where unsupported types cause the API request to fail.
+
+**Rationale**: Silent drops are a bug — the user sends a file, the LLM never sees it, and nobody gets feedback. With errors, either the client validates beforehand (preferred) or the user sees an explicit failure. All three providers behave identically: unsupported = fail.
+
+**Supported types per adapter (adk-utils-go v0.3.1)**:
+
+| Type | Gemini | OpenAI | Anthropic |
+|---|---|---|---|
+| Images (JPEG, PNG, GIF, WebP) | ✅ (native) | ✅ (data URI) | ✅ (Base64ImageSource) |
+| PDF | ✅ (native) | ✅ (FileParam) | ✅ (Base64PDFSource) |
+| Text (text/*) | ✅ (native) | ✅ (FileParam) | ✅ (PlainTextSource) |
+| Audio (WAV, MP3, WebM) | ✅ (native) | ✅ (InputAudio) | ❌ error |
+| Video, other | ✅ (native) | ❌ error | ❌ error |
+
+**Do not**: Silently drop unsupported `InlineData` parts. Do not convert them to text descriptions. Return `fmt.Errorf("unsupported inline data MIME type for %s: %s")`.
+
+**Files**: `adk-utils-go/genai/openai/openai.go` (`convertInlineDataToPart`), `adk-utils-go/genai/anthropic/anthropic.go` (`convertInlineDataToBlock`).

.agents/TODO.md

@@ -1,26 +1,29 @@
 # Magec - TODO
 
-## High Priority
-
-### Large Message Handling in Telegram and Slack
+## ~~Large Message Handling in Telegram and Slack~~ ✅
 
-**Problem**: No validation on inbound message size from Telegram/Slack, and outbound responses to Telegram may exceed the 4096-character message limit. Large inputs could cause excessive memory usage or unexpected behavior, and oversized responses will fail silently or get truncated by the API.
-
-**Solution**:
-- **Inbound**: Add a max input length check in both clients. Reject or truncate messages that exceed a reasonable threshold (e.g. 16K chars) with a user-friendly error.
-- **Outbound (Telegram)**: Split responses exceeding 4096 chars into multiple sequential messages. Preserve markdown formatting across splits where possible.
-- **Outbound (Slack)**: Slack's limit is ~40K per message block — less urgent but should still have a safety check.
-
-**Modify**: `server/clients/telegram/bot.go`, `server/clients/slack/bot.go`
+Implemented. See `server/clients/msgutil/` package.
 
 ---
 
+## High Priority
+
 ### Multimodal File/Image Support in Clients
 
 **Problem**: Telegram and Slack clients only handle text and voice messages. Users sending images, documents, PDFs, or other files get silently ignored.
 
 **Solution**: Download files from Telegram/Slack, encode as base64, and send as `inlineData` parts alongside text in the ADK `/run` request. The ADK already supports `genai.Part{InlineData: &Blob{Data, MIMEType}}` — zero backend changes needed.
 
+**Adapter support (adk-utils-go v0.3.1)**:
+- **Gemini**: passes all `InlineData` transparently to the API. Unsupported types are rejected by Google's API.
+- **OpenAI**: translates images (JPEG, PNG, GIF, WebP), audio (WAV, MP3, MPEG, WebM), and files (PDF, text/*). Unsupported types return an error.
+- **Anthropic**: translates images (JPEG, PNG, GIF, WebP), PDFs, and text documents (text/*). Unsupported types return an error.
+- All three adapters behave the same: if a MIME type can't be translated, the request fails. No silent drops.
+
+**File size limits**: 5MB per file, 10MB total per message, max 10 files per message. Validated client-side before download.
+
+**Supported types (denominator común)**: JPEG, PNG, GIF, WebP. PDF and text/* work on Gemini + Anthropic. Audio works on Gemini + OpenAI.
+
 **Telegram** (`server/clients/telegram/bot.go`):
 - Current state: only `Voice` (dedicated handler) and `Text` (predicate at ~line 171 requires `Text != ""` and `Voice == nil`). Everything else is silently dropped.
 - Add handler for `Document`, `Photo`, `Video`, `Audio`, `Animation`, `VideoNote`, `Sticker`. All have `FileID` → `bot.GetFile()` → download bytes.
@@ -46,7 +49,7 @@
 }
 ```
 
-**File size validation**: Add 20MB limit (denominator común: Gemini 20MB, OpenAI 20MB, Anthropic 5MB for images). Telegram API limits bots to 20MB anyway. Reject oversized files with user-friendly message.
+**File size validation**: 5MB per file, 10MB total per message, max 10 files. Reject oversized files with user-friendly message.
 
 **LLM limitations**: GPT-4o/Claude/Gemini handle images and PDFs natively. For Word/Excel/CSV, the model may not support them — the user gets a natural "I can't process this format" response from the LLM itself.
 
@@ -127,28 +130,9 @@ See `.agents/ADK_TOOLS.md` for protocol details.
 
 ---
 
-### Artifact Management Toolset
-
-**Problem**: ADK has artifact storage (versioned, session-scoped) and REST endpoints for clients to download them, but the LLM has no way to create, read, list, or delete artifacts. Without tools that call `ctx.SaveArtifact()` / `ctx.LoadArtifact()`, the artifact system is dead weight.
-
-**Solution**: Build a base toolset with four Go-native tools using `functiontool`:
-- `save_artifact(name, content, mimeType)` — saves content as a versioned artifact in the session
-- `load_artifact(name)` — reads an artifact (latest version) back into context
-- `list_artifacts()` — lists all artifacts in the current session
-- `delete_artifact(name)` — removes an artifact
-
-**Use cases**:
-- LLM generates a report/export → `save_artifact()` → user downloads via Voice UI / Telegram / Slack using existing ADK GET endpoints
-- Flow pipelines: step 1 produces data → `save_artifact()`, step 2 reads → `load_artifact()` and transforms
-- Combined with a filesystem MCP: `load_artifact()` → process → `write_file()` to persist externally, or `read_file()` → `save_artifact()` to make available for download
-
-**Design**:
-- Configurable per agent (not all agents need it) — toggle in agent config, similar to memory tools
-- Sandboxed by session — no security risk, no file system access
-- ADK handles versioning and storage automatically
-- Replaces `loadartifactstool` from ADK (read-only) with a complete CRUD toolset
+### ~~Artifact Management Toolset~~ ✅
 
-**Modify**: `server/agent/agent.go` (register toolset in `buildToolsets`), new file `server/agent/tools/artifacts.go`, `server/store/types.go` (agent config toggle), `frontend/admin-ui/` (agent form toggle)
+Implemented. See `server/agent/tools/artifacts/toolset.go` — provides `save_artifact`, `load_artifact`, and `list_artifacts` tools via `functiontool.New`. Supports text and base64 binary content. Wired into `base_toolset.go` so all agents get it. Filesystem-backed via `adk-utils-go/artifact/filesystem` (persists across restarts). Clients (Telegram and Slack) auto-deliver new artifacts as file attachments after each `/run` response using before/after diff of the artifact list REST endpoint.
 
 ---
 

server/agent/agent.go

@@ -46,6 +46,7 @@ import (
 	memorypostgres "github.com/achetronic/adk-utils-go/memory/postgres"
 	sessionredis "github.com/achetronic/adk-utils-go/session/redis"
 	toolsmemory "github.com/achetronic/adk-utils-go/tools/memory"
+	artifactfs "github.com/achetronic/adk-utils-go/artifact/filesystem"
 
 	"github.com/achetronic/magec/server/config"
 	"github.com/achetronic/magec/server/contextwindow"
@@ -71,6 +72,14 @@ When a user asks you to remember something or asks about past information:
 
 When a user shares preferences or important information, proactively save it to memory for future reference.`
 
+const artifactInstruction = `
+You have access to artifact tools for creating and managing files:
+- Use 'save_artifact' to save code, documents, data files, or any content that should be delivered as a downloadable file. Provide a filename (e.g. "report.md", "main.py", "data.csv"), the content, and optionally a mime_type. For binary content, set is_base64=true and provide base64-encoded data.
+- Use 'load_artifact' to retrieve a previously saved artifact by name.
+- Use 'list_artifacts' to see all artifacts in the current session.
+
+IMPORTANT: When generating code files, long documents, configuration files, scripts, or any substantial structured content, ALWAYS use save_artifact instead of pasting it in the chat. The artifact will be delivered to the user as a downloadable file automatically.`
+
 // Service wraps the ADK REST handler that serves all configured agents.
 // Incoming requests are routed to the correct agent by the appName field.
 type Service struct {
@@ -128,6 +137,13 @@ func New(ctx context.Context, agents []store.AgentDefinition, backends []store.B
 	// Rebuilt from scratch on every hot-reload (store change).
 	llmMap := make(map[string]model.LLM, len(agents))
 
+	artifactSvc, err := artifactfs.NewFilesystemService(artifactfs.FilesystemServiceConfig{
+		BasePath: filepath.Join("data", "artifacts"),
+	})
+	if err != nil {
+		return nil, fmt.Errorf("artifact service: %w", err)
+	}
+
 	baseTset, err := newBaseToolset()
 	if err != nil {
 		return nil, fmt.Errorf("failed to create base toolset: %w", err)
@@ -195,8 +211,9 @@ func New(ctx context.Context, agents []store.AgentDefinition, backends []store.B
 	}
 
 	launcherCfg := &launcher.Config{
-		SessionService: sessionSvc,
-		AgentLoader:    loader,
+		SessionService:  sessionSvc,
+		AgentLoader:     loader,
+		ArtifactService: artifactSvc,
 	}
 	if memorySvc != nil {
 		launcherCfg.MemoryService = memorySvc
@@ -519,6 +536,8 @@ func buildInstruction(agentDef store.AgentDefinition, mcpServerMap map[string]st
 		instruction += memoryInstruction
 	}
 
+	instruction += artifactInstruction
+
 	for _, mcpName := range agentDef.MCPServers {
 		if srv, ok := mcpServerMap[mcpName]; ok && srv.SystemPrompt != "" {
 			instruction += "\n\n" + srv.SystemPrompt

server/agent/base_toolset.go

@@ -1,28 +1,42 @@
 package agent
 
 import (
+	"fmt"
+
 	"google.golang.org/adk/agent"
 	"google.golang.org/adk/tool"
+
+	toolsartifacts "github.com/achetronic/magec/server/agent/tools/artifacts"
 )
 
-// baseToolset provides tools that are available to every agent regardless of
-// configuration.
-//
-// TODO: Explore injecting exit_loop only to agents inside a loopagent (option 3).
-// This would require cloning agents when building flow steps so the same agent
-// definition can participate in a loop (with exit_loop) and outside one (without).
 type baseToolset struct {
-	tools []tool.Tool
+	tools         []tool.Tool
+	artifactTools *toolsartifacts.Toolset
 }
 
 func newBaseToolset() (*baseToolset, error) {
-	return &baseToolset{tools: []tool.Tool{}}, nil
+	artifactTs, err := toolsartifacts.NewToolset()
+	if err != nil {
+		return nil, fmt.Errorf("failed to create artifact toolset: %w", err)
+	}
+
+	return &baseToolset{
+		tools:         []tool.Tool{},
+		artifactTools: artifactTs,
+	}, nil
 }
 
 func (b *baseToolset) Name() string {
 	return "base_toolset"
 }
 
-func (b *baseToolset) Tools(_ agent.ReadonlyContext) ([]tool.Tool, error) {
-	return b.tools, nil
+func (b *baseToolset) Tools(ctx agent.ReadonlyContext) ([]tool.Tool, error) {
+	artTools, err := b.artifactTools.Tools(ctx)
+	if err != nil {
+		return b.tools, nil
+	}
+	all := make([]tool.Tool, 0, len(b.tools)+len(artTools))
+	all = append(all, b.tools...)
+	all = append(all, artTools...)
+	return all, nil
 }