feat: s3 storage and richer file support (cgoinglove#301)

## Summary

- Stand up a production-ready S3 storage driver: presigned PUT upload
flow, head/tail download helpers, CLI smoke check, new env surface area,
and doc updates for
  teams migrating off Vercel Blob.
- Rebuild the upload UX into a thread-scoped, multi-file pipeline with
drag-and-drop overlay, retry-aware queueing, per-file progress, and
consistent metadata fan-
  out for chat state, clipboard, and API payloads.
- Normalize attachment rendering so user vs. assistant bubbles style
correctly, filenames truncate, filetype badges surface, and download
affordances remain obvious
  —even when multiple cards stack together.
- Tighten the file-handling contract: only image/PDF MIME types flow
through as file parts; everything else gracefully downgrades to a
source-url card so models
never receive unsupported media. CSV is special-cased with a hidden
markdown preview that the model consumes while the UI stays clean.
- Advertise per-model MIME capabilities through customModelProvider,
letting the composer warn early and making unsupported models fall back
without breaking tool
  calls.
- Auto-ingest CSV attachments by streaming server-side previews,
plugging into chat history so agents can immediately cite tabular
context. Added coverage for
  preview formatting and ingestion gating.
- Document the new storage story (docs/storage/s3-setup.md) and codex
agent metadata (AGENTS.md), plus extend unit tests across storage utils,
ingestion, and MIME
  support.

### File Handling Model

- Provider MIME gates: The client now trusts explicit allowlists
attached to each model. If a provider (e.g., OpenAI) cannot ingest a
media type, we flip the part
to source-url so the LLM sees a link instead of throwing “functionality
not supported.” This keeps uploads future-proof and highlights the
narrow set of formats
  we genuinely support today (images + PDF).
- CSV ingestion preview: CSV files render an ingestionPreview text part
that the UI suppresses but the model consumes, giving instant structured
context without
  polluting the chat transcript.
- Threaded uploader: Each thread maintains its own queue, enabling
parallel uploads, progress tracking, and safe retry semantics.
Drag-and-drop, button uploads, and
clipboard pastes all flow through a single hook so we don’t
double-handle state.
- UI parity: File bubbles respect author context, maintain readability
with truncation and badges, and keep download buttons consistent even
when stacking multiple
  attachments.

## Screenshots / Recordings

### Drag and Drop

https://github.com/user-attachments/assets/1346b0e7-fa86-4a8f-abee-bd69542c93b9

### Multi File Drag and Drop

https://github.com/user-attachments/assets/d3e3bd27-6cee-4ac4-86f6-b08bc1ce5973

### CSV Preview

https://github.com/user-attachments/assets/9382005f-ea50-4377-9e15-34d3bb88c7fc

## Configuration Notes

- .env.example now documents FILE_STORAGE_TYPE (vercel-blob or s3),
FILE_STORAGE_S3_BUCKET, region, optional CDN origin, and other S3 knobs.
- S3 driver uses AWS credentials from env or the default provider chain;
fallback instructions live in docs/storage/s3-setup.md.
- CSV ingestion continues to leverage /api/storage/ingest; ensure your
storage backend serves direct downloads for the preview step.

## Verification Guide

1. pnpm dev
2. Upload a mix of image + CSV + PDF via the picker and via
drag-and-drop. Confirm badges, truncation, download button, and
per-thread queues.
3. Switch to a model that doesn’t support file parts; ensure attachments
render as “source-url” cards instead of failing tool calls.
4. Send a CSV attachment and validate the assistant references the
preview immediately with no visible summary text in the chat.
5. (Optional) Point to S3 credentials, generate a presigned PUT (CLI
script provided), upload through the UI, and confirm the file is present
via S3 console or aws
    s3 head object.

## Tests

- pnpm format
- pnpm lint
- pnpm check-types
- pnpm test

## Documentation & Follow-up

- New S3 setup guide at docs/storage/s3-setup.md.
- Added AGENTS.md for Codex agent metadata.
- Future: add XLSX ingestion once a server-side parser/vectorization
pipeline lands; capture upload telemetry once S3 analytics hooks are in
place.

---------

Co-authored-by: Copilot <[email protected]>

Author: mrjasonroy

.env.example

@@ -102,8 +102,25 @@ MCP_MAX_TOTAL_TIMEOUT=
 # BLOB_READ_WRITE_TOKEN=
 
 
-# -- S3 (planned driver) --
+# -- S3 --
 # FILE_STORAGE_TYPE=s3
 # FILE_STORAGE_PREFIX=uploads
 # FILE_STORAGE_S3_BUCKET=
 # FILE_STORAGE_S3_REGION=
+# Optional: Use when serving files via CDN/custom domain
+# FILE_STORAGE_S3_PUBLIC_BASE_URL=https://cdn.example.com
+# Optional: For S3-compatible endpoints (e.g., MinIO)
+# FILE_STORAGE_S3_ENDPOINT=http://localhost:9000
+# Optional: Force path-style URLs (1/true to enable)
+# FILE_STORAGE_S3_FORCE_PATH_STYLE=1
+
+
+# AWS Credentials (server only)
+# The AWS SDK automatically discovers credentials in this order:
+# 1) Environment variables below, 2) ~/.aws/credentials or AWS_PROFILE,
+# 3) IAM role attached to the runtime (EC2/ECS/EKS/Lambda).
+# You do NOT need to set these when using an IAM role.
+# AWS_ACCESS_KEY_ID=
+# AWS_SECRET_ACCESS_KEY=
+# AWS_SESSION_TOKEN=
+# AWS_REGION=us-east-1

AGENTS.md

@@ -0,0 +1,42 @@
+# Repository Guidelines
+
+## Project Structure & Module Organization
+- App code lives in `src`.
+  - `src/app` (Next.js routes, API, middleware)
+  - `src/components` (UI; reusable components in PascalCase)
+  - `src/lib` (helpers: auth, db, ai, validations, etc.)
+  - `src/hooks` (React hooks: `useX`)
+- Assets in `public/`. End‑to‑end tests in `tests/`. Scripts in `scripts/`. Docker files in `docker/`.
+
+## Build, Test, and Development Commands
+- `pnpm dev` — Run the app locally (Next.js dev server).
+- `pnpm build` / `pnpm start` — Production build and run.
+- `pnpm lint` / `pnpm lint:fix` — ESLint + Biome checks and autofix.
+- `pnpm format` — Format with Biome.
+- `pnpm test` / `pnpm test:watch` — Unit tests (Vitest).
+- `pnpm test:e2e` — Playwright tests; uses `playwright.config.ts` webServer.
+- DB: `pnpm db:push`, `pnpm db:studio`, `pnpm db:migrate` (Drizzle Kit).
+- Docker: `pnpm docker-compose:up` / `:down` to run local stack.
+
+## Coding Style & Naming Conventions
+- TypeScript everywhere. Prefer `zod` for validation.
+- Formatting via Biome: 2 spaces, LF, width 80, double quotes.
+- Components: `PascalCase.tsx`; hooks/utilities: `camelCase.ts`.
+- Co-locate small module tests next to code; larger suites under `tests/`.
+- Keep modules focused; avoid circular deps; use `src/lib` for shared logic.
+
+## Testing Guidelines
+- Unit tests: Vitest, filename `*.test.ts(x)`.
+- E2E: Playwright under `tests/`, filename `*.spec.ts`.
+- Run locally: `pnpm test` and `pnpm test:e2e` (ensure app is running or let Playwright start via config).
+- Add tests for new features and bug fixes; cover happy path + one failure mode.
+
+## Commit & Pull Request Guidelines
+- Conventional Commits: `feat:`, `fix:`, `chore:`, `docs:`, etc. Example: `feat: add image generation tool`.
+- Branch names: `feat/…`, `fix/…`, `chore/…`.
+- PRs: clear description, linked issues, screenshots or terminal output when UI/CLI changes; list test coverage and manual steps.
+- Before opening PR: `pnpm check` (lint+types+tests) should pass.
+
+## Security & Configuration Tips
+- Copy `.env.example` to `.env`; never commit secrets. For local HTTP use `NO_HTTPS=1` or `pnpm build:local`.
+- If using DB/Redis locally, start services via Docker scripts or your own stack.

docs/storage/s3-setup.md

@@ -0,0 +1,61 @@
+# S3 Storage Setup
+
+This app supports S3 for file uploads (dev/prod). Development can rely on presigned PUTs directly from the browser, while production should keep the bucket private and serve via CDN (CloudFront + Origin Access Control) or signed GET URLs.
+
+## Buckets
+- Pick a region (e.g., `us-east-2`)
+  - Dev/Test example: `better-chatbot-dev` (public GET on `uploads/` only if needed)
+  - Prod example: `better-chatbot-prod` (private)
+- Enable default encryption (SSE-S3) and versioning on both buckets.
+
+## CORS
+- Dev bucket: allow PUT/GET/HEAD from the origins you use locally and in staging, for example:
+  - `http://localhost:3000`, `http://127.0.0.1:3000`
+  - `https://staging.your-domain.com`, `http://staging.your-domain.com`
+- Prod bucket: allow GET/HEAD only from your production domain (e.g., `https://app.your-domain.com`). Avoid enabling browser PUT in production.
+
+## Dev public-read policy (prefix-only)
+Grant public GET for the `uploads/` prefix on the dev bucket only if you need unauthenticated downloads:
+```
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "AllowPublicReadForUploadsPrefix",
+      "Effect": "Allow",
+      "Principal": "*",
+      "Action": "s3:GetObject",
+      "Resource": "arn:aws:s3:::better-chatbot-dev/uploads/*"
+    }
+  ]
+}
+```
+
+## IAM (app runtime)
+Least privilege for app role/user:
+- Actions: `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`, `s3:HeadObject`
+- Resources: `arn:aws:s3:::<bucket-name>/uploads/*`
+
+## Env configuration
+- Dev/local:
+  - `FILE_STORAGE_TYPE=s3`
+  - `FILE_STORAGE_PREFIX=uploads`
+  - `FILE_STORAGE_S3_BUCKET=better-chatbot-dev`
+  - `FILE_STORAGE_S3_REGION=us-east-2` (or set `AWS_REGION`)
+  - Use AWS SSO/profile or `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`
+- Prod:
+  - `FILE_STORAGE_S3_BUCKET=better-chatbot-prod`
+  - Prefer CloudFront; set `FILE_STORAGE_S3_PUBLIC_BASE_URL=https://<cdn-domain>`
+
+## Verify locally
+- Ensure `aws sso login --profile <your_profile>` (or credentials are already available).
+- Test presign script:
+```
+AWS_PROFILE=<your_profile> \
+FILE_STORAGE_TYPE=s3 \
+FILE_STORAGE_S3_BUCKET=better-chatbot-dev \
+FILE_STORAGE_S3_REGION=us-east-2 \
+pnpm tsx scripts/verify-s3-upload-url.ts
+```
+- You should get `{ directUploadSupported: true, url, key, method: PUT }`.
+- Upload with curl (optional): `curl -X PUT -H "Content-Type: image/png" --data-binary @file.png "<url>"`.

messages/en.json

@@ -194,7 +194,7 @@
   "Chat": {
     "Error": "Chat Error",
     "thisMessageWasNotSavedPleaseTryTheChatAgain": "This message was not saved. Please try the chat again.",
-    "uploadImage": "Upload Image",
+    "uploadImage": "Upload File",
     "generateImage": "Generate Image",
     "imageUploadedSuccessfully": "Image uploaded successfully",
     "pleaseUploadImageFile": "Please upload an image file",

messages/es.json

@@ -74,7 +74,7 @@
   "Chat": {
     "Error": "Error de Chat",
     "thisMessageWasNotSavedPleaseTryTheChatAgain": "Este mensaje no se guardó. Por favor, intenta el chat nuevamente.",
-    "uploadImage": "Subir Imagen",
+    "uploadImage": "Subir archivo",
     "generateImage": "Generar Imagen",
     "imageUploadedSuccessfully": "Imagen subida exitosamente",
     "pleaseUploadImageFile": "Por favor sube un archivo de imagen",

messages/fr.json

@@ -74,7 +74,7 @@
   "Chat": {
     "Error": "Erreur de Chat",
     "thisMessageWasNotSavedPleaseTryTheChatAgain": "Ce message n'a pas été enregistré. Veuillez réessayer le chat.",
-    "uploadImage": "Télécharger une Image",
+    "uploadImage": "Téléverser un fichier",
     "generateImage": "Générer une Image",
     "imageUploadedSuccessfully": "Image téléchargée avec succès",
     "pleaseUploadImageFile": "Veuillez télécharger un fichier image",

messages/ja.json

@@ -74,7 +74,7 @@
   "Chat": {
     "Error": "チャットエラー",
     "thisMessageWasNotSavedPleaseTryTheChatAgain": "このメッセージは保存されませんでした。もう一度チャットをお試しください。",
-    "uploadImage": "画像をアップロード",
+    "uploadImage": "ファイルをアップロード",
     "generateImage": "画像を生成",
     "imageUploadedSuccessfully": "画像が正常にアップロードされました",
     "pleaseUploadImageFile": "画像ファイルをアップロードしてください",

messages/ko.json

@@ -75,7 +75,7 @@
   "Chat": {
     "Error": "채팅 오류",
     "thisMessageWasNotSavedPleaseTryTheChatAgain": "이 메시지는 저장되지 않았습니다. 다시 시도해주세요.",
-    "uploadImage": "이미지 업로드",
+    "uploadImage": "파일 업로드",
     "generateImage": "이미지 만들기",
     "imageUploadedSuccessfully": "이미지가 성공적으로 업로드되었습니다",
     "pleaseUploadImageFile": "이미지 파일을 업로드해주세요",

messages/zh.json

@@ -75,7 +75,7 @@
   "Chat": {
     "Error": "聊天错误",
     "thisMessageWasNotSavedPleaseTryTheChatAgain": "此消息未保存。请重试聊天。",
-    "uploadImage": "上传图片",
+    "uploadImage": "上传文件",
     "generateImage": "生成图片",
     "imageUploadedSuccessfully": "图片上传成功",
     "pleaseUploadImageFile": "请上传图片文件",

src/app/api/chat/route.ts

@@ -49,6 +49,8 @@ import { colorize } from "consola/utils";
 import { generateUUID } from "lib/utils";
 import { nanoBananaTool, openaiImageTool } from "lib/ai/tools/image";
 import { ImageToolName } from "lib/ai/tools";
+import { buildCsvIngestionPreviewParts } from "@/lib/ai/ingest/csv-ingest";
+import { serverFileStorage } from "lib/file-storage";
 
 const logger = globalLogger.withDefaults({
   message: colorize("blackBright", `Chat API: `),
@@ -72,6 +74,7 @@ export async function POST(request: Request) {
       allowedMcpServers,
       imageTool,
       mentions = [],
+      attachments = [],
     } = chatApiSchemaRequestBodySchema.parse(json);
 
     const model = customModelProvider.getModel(chatModel);
@@ -104,6 +107,70 @@ export async function POST(request: Request) {
     if (messages.at(-1)?.id == message.id) {
       messages.pop();
     }
+    const ingestionPreviewParts = await buildCsvIngestionPreviewParts(
+      attachments,
+      (key) => serverFileStorage.download(key),
+    );
+    if (ingestionPreviewParts.length) {
+      const baseParts = [...message.parts];
+      let insertionIndex = -1;
+      for (let i = baseParts.length - 1; i >= 0; i -= 1) {
+        if (baseParts[i]?.type === "text") {
+          insertionIndex = i;
+          break;
+        }
+      }
+      if (insertionIndex !== -1) {
+        baseParts.splice(insertionIndex, 0, ...ingestionPreviewParts);
+        message.parts = baseParts;
+      } else {
+        message.parts = [...baseParts, ...ingestionPreviewParts];
+      }
+    }
+
+    if (attachments.length) {
+      const firstTextIndex = message.parts.findIndex(
+        (part: any) => part?.type === "text",
+      );
+      const attachmentParts: any[] = [];
+
+      attachments.forEach((attachment) => {
+        const exists = message.parts.some(
+          (part: any) =>
+            part?.type === attachment.type && part?.url === attachment.url,
+        );
+        if (exists) return;
+
+        if (attachment.type === "file") {
+          attachmentParts.push({
+            type: "file",
+            url: attachment.url,
+            mediaType: attachment.mediaType,
+            filename: attachment.filename,
+          });
+        } else if (attachment.type === "source-url") {
+          attachmentParts.push({
+            type: "source-url",
+            url: attachment.url,
+            mediaType: attachment.mediaType,
+            title: attachment.filename,
+          });
+        }
+      });
+
+      if (attachmentParts.length) {
+        if (firstTextIndex >= 0) {
+          message.parts = [
+            ...message.parts.slice(0, firstTextIndex),
+            ...attachmentParts,
+            ...message.parts.slice(firstTextIndex),
+          ];
+        } else {
+          message.parts = [...message.parts, ...attachmentParts];
+        }
+      }
+    }
+
     messages.push(message);
 
     const supportToolCall = !isToolCallUnsupportedModel(model);