Skip to content

Commit 384c882

Browse files
authored
feat(copilot): voice input (STT + BYOK), improved composer, and config-driven demo (#36)
## Summary Adds **voice input** to SimplePDF Copilot and reworks the composer + demo entitlement model. All changes are under `copilot/`. ## Changes - **Speech-to-text** in the composer: tap the mic → record → transcribe on stop → editable transcript. Server `/api/transcribe` with a streaming SSE relay; capped, metered, and sanitized (no raw provider text to the client). - **BYOK speech-to-text**: a per-function model picker with **Chat** and **Speech-to-Text** tabs. BYOK transcription is **browser-direct** (OpenAI or any OpenAI-compatible endpoint), with an AES-GCM credential vault, Web-Locks-serialized mutations, linearizable revocation, a custom-URL SSRF/loopback policy, and format-matched save-time validation. - **Improved composer**: one rounded box with **no layout shift** on mic-click (the textarea ↔ a recipient-aware recording prompt swap; mic+send ↔ the recording controls), a square-rounded blue send button, and 12px message bubbles. A voice rate-limit surfaces the same demo-limit + share panel as chat. - **Config-driven demo mode** (replaces the old `?share=` invite system): demo mode is on iff `DEMO_CHAT_API_KEY` + `DEMO_CHAT_MODEL` + `DEMO_RATE_LIMIT_TURNS` + `DEMO_STT_OPENAI_API_KEY` are all set. Every visitor then uses the operator's keys, rate-limited **per IP** (no invite links). Removed the `SHARED_API_KEYS` map, `share_query.ts`, and all client `?share=` plumbing. - **System prompt**: refactored the form-filling prompt (collapsed overlapping interactivity/filling/tone sections, trimmed verbose examples) and added a **bulk-fill branch** — when the user hands over many values at once, fill them all, confirm once, then resume the one-at-a-time flow. - **Security**: `ChatRequestSchema` now allowlists `user`/`assistant` roles, so a `role:'system'` message in the request body can no longer be promoted into the system prompt (system-authority prompt injection). Found via an adversarial review. ## Reviews & tests Full-branch and per-change `/code-review`, including BADobe security passes on the transcribe route, the BYOK vault/revocation, the demo entitlement cutover, and the chat injection vector. **168 unit tests, `tsc` + Biome green; 23 locales complete.** ## Operator note Demo mode is now **open to anyone who can reach the page** (no invite gating), so the per-IP turn cap (`DEMO_RATE_LIMIT_TURNS`) is the cost control — size it accordingly and make sure your edge sets a non-spoofable client-IP header. `.env.example` documents all four `DEMO_*` vars. ## Deferred follow-ups (non-blocking) Real-mic / cross-browser visual QA, a two-page Playwright revocation-race e2e, format-matched fixture speech clips, and ~17 pre-existing en-only `modelPicker` i18n keys.
1 parent fd243f0 commit 384c882

97 files changed

Lines changed: 6401 additions & 1217 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

copilot/README.md

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,10 @@ Browser
6969
- SimplePDF Copilot drives the editor through `postMessage` (focus a field, set a value, navigate, submit)
7070
- LLM streaming runs through your server via the Vercel AI SDK; you choose the provider
7171
- Tool calls are executed in the browser, against the iframe. Your server only proxies the chat stream.
72+
- **Voice input is different — it is a deliberate audio egress, on one of two paths.** Dictating into the composer records a short audio clip in the browser; recording starts when you tap the mic, and when you confirm it (✓) the clip is transcribed and the editable transcript drops into the textarea. Two routes, each named in the recorder before the audio is sent:
73+
- **Demo (server):** when the deployment is in demo mode (operator keys configured), the clip uploads to `/api/transcribe`, which forwards it to OpenAI (`gpt-4o-transcribe`) and returns the transcript. So **audio leaves the browser to SimplePDF's server and then OpenAI** (the server keeps no audio, logs no transcript).
74+
- **BYOK (browser-direct):** configure a Speech-to-Text provider (OpenAI or a custom OpenAI-compatible endpoint) in the model picker's Speech-to-Text tab; the clip is sent **directly from the browser to that endpoint, never to SimplePDF**. The key lives only in this browser's encrypted vault — a demo/reference feature (a browser-held key is exposed to anything on the page).
75+
- In both cases PDF bytes still stay on-device, and audio is sent only when you confirm the recording (never automatically). The recorder prompt names the actual audio recipient before you confirm: the **demo** prompt reads "Speak to SimplePDF Copilot…" (its server→OpenAI flow is the one described above, and "What is this demo?" documents it); the **BYOK** prompt reads "Speak to OpenAI…" or "Speak to <your endpoint>…" (sent directly to that provider, not to SimplePDF).
7276

7377
## Built with
7478

@@ -108,16 +112,30 @@ In the running app, open the chat sidebar, click **Bring your own provider**, pa
108112
> [!IMPORTANT]
109113
> **Keep the dev port at 3001.** The SimplePDF demo workspace whitelists exactly one local origin, `http://localhost:3001`, and the editor will only load on a parent page served from that exact host and port. The browser enforces this on iframe load: any other port (e.g. 3000, 5173) or any other host is refused. The `dev` script in `package.json` pins port 3001; don't override it with `--port` flags. To run on your own domain or a different port, you need a SimplePDF [Pro](https://simplepdf.com/pricing) account so you can set your own `companyIdentifier` and whitelist your origin in the SimplePDF dashboard.
110114
111-
### Share it without asking viewers for a key
115+
### Run the demo without asking viewers for a key
112116

113-
Sharing the demo with non-technical users (a teammate, a prospect, a friend) is friction-heavy if every visitor has to paste a provider key. To skip that step, set `SHARED_API_KEYS` in your `.env` and append `?share=<id>` to the URL: the server pays for the LLM under your account, the chat opens already wired up, and the Model Picker stays out of the way.
117+
Asking every visitor to paste a provider key is friction-heavy. To skip that, put the deployment **in demo mode**: configure a single chat key/model/turn-cap plus a transcription key in your `.env`, and the server pays for chat + voice under your account for every visitor. There are no invite links — demo mode is simply on whenever both are configured. The chat opens already wired up and the Model Picker stays out of the way (visitors can still bring their own key to override).
114118

115-
Two providers are supported on the shared-key path:
119+
Demo mode requires **both**:
116120

117-
- Anthropic Claude Haiku 4.5 (`model: "anthropic_haiku_4_5"`)
118-
- DeepSeek V4 Flash (`model: "deepseek_v4_flash"`)
121+
- `DEMO_CHAT_API_KEY` + `DEMO_CHAT_MODEL` + `DEMO_RATE_LIMIT_TURNS` — the chat key, the model, and the per-IP turn cap.
122+
- `DEMO_STT_OPENAI_API_KEY` — the voice transcription key (transcription-only, never your chat key).
119123

120-
See [`.env.example`](./.env.example) for the JSON shape, the per-share rate-limit options, and the portable base64 one-liner for hosts that mangle embedded quotes (DigitalOcean App Platform, Render, fly.io). Then visit `http://localhost:3001/?share=<id>` and you're set.
124+
Two chat models are supported on the demo path:
125+
126+
- Anthropic Claude Haiku 4.5 (`DEMO_CHAT_MODEL=anthropic_haiku_4_5`)
127+
- DeepSeek V4 Flash (`DEMO_CHAT_MODEL=deepseek_v4_flash`)
128+
129+
Leave the demo vars unset and the deployment runs **BYOK-only**: every visitor brings their own key via the Model Picker. Note: with no invite gating, demo mode is open to anyone who can reach the page, so the **per-IP turn cap (`DEMO_RATE_LIMIT_TURNS`) is the cost control** — size it accordingly. See [`.env.example`](./.env.example) for the exact vars.
130+
131+
### Voice input (dictation)
132+
133+
The chat composer shows a microphone whenever the browser can record. Clicking it needs **both** a Chat model and a Speech-to-Text provider configured (a transcript you can't send is useless), so it opens the model picker on whichever is missing, else it records. Two transcription routes:
134+
135+
- **Demo (server):** when the deployment is in demo mode (the demo vars above are set, including `DEMO_STT_OPENAI_API_KEY`), the clip is transcribed via `/api/transcribe` on the operator's key. Voice and chat share the same demo entitlement, so both require the keys to be configured; without the transcription key the deployment isn't in demo mode and voice + demo chat are unavailable (BYOK still works).
136+
- **BYOK (browser-direct):** in the picker's **Speech-to-Text** tab, configure OpenAI (`gpt-4o-mini-transcribe` / `gpt-4o-transcribe`) or a custom OpenAI-compatible endpoint. The clip is transcribed directly browser→provider, never touching SimplePDF's server. No env var needed.
137+
138+
See the privacy notes above for the per-route audio-egress disclosure.
121139

122140
### Load a specific document via `?url=`
123141

@@ -129,7 +147,7 @@ http://localhost:3001/?url=https%3A%2F%2Fdemo.simplepdf.com%2Fdocuments%2Fc28f06
129147

130148
URL-encode the value whenever it carries its own query string (e.g. `?prefill=`), otherwise the nested params get parsed as part of the page URL and dropped.
131149

132-
The value must be an absolute `http(s)` URL on your configured base-domain family (`*.simplepdf.com` by default, or whatever host `VITE_SIMPLEPDF_BASE_DOMAIN` resolves to). Third-party origins are rejected on purpose: the iframe is granted clipboard access and is wired to the `postMessage` bridge, so framing an arbitrary site would hand it both. A malformed or off-domain `?url=` silently falls back to the default demo form. `?url=` combines with `?lang=` and `?share=`; when set, it wins for what the editor loads.
150+
The value must be an absolute `http(s)` URL on your configured base-domain family (`*.simplepdf.com` by default, or whatever host `VITE_SIMPLEPDF_BASE_DOMAIN` resolves to). Third-party origins are rejected on purpose: the iframe is granted clipboard access and is wired to the `postMessage` bridge, so framing an arbitrary site would hand it both. A malformed or off-domain `?url=` silently falls back to the default demo form. `?url=` combines with `?lang=`; when set, it wins for what the editor loads.
133151

134152
### Ship it on your own domain
135153

@@ -158,9 +176,10 @@ For multi-container deployments (or any deploy where you want per-IP rate-limit
158176
The button reads [`.do/deploy.template.yaml`](https://github.com/SimplePDF/simplepdf-embed/blob/main/.do/deploy.template.yaml) at the repo root: Node 24 buildpack, single instance, builds from `/copilot`. DigitalOcean prompts you for the env vars at setup time:
159177

160178
- `VITE_SIMPLEPDF_COMPANY_IDENTIFIER` (required, no default): your SimplePDF company subdomain (Pro plan or higher)
161-
- `SHARED_API_KEYS` (optional secret): paste a JSON or base64 payload to enable the `?share=<id>` flow; leave empty for BYOK-only
179+
- `DEMO_CHAT_API_KEY` / `DEMO_CHAT_MODEL` / `DEMO_RATE_LIMIT_TURNS` + `DEMO_STT_OPENAI_API_KEY` (optional secrets): set **all** of them to put the deployment in demo mode (the server pays for chat + voice, capped per IP by the turn count); leave unset for BYOK-only
162180
- `REDIS_URL` (optional secret): a Redis-protocol connection URL (Valkey on DO Managed Caching works as-is). Required for multi-container deployments where per-IP rate-limit counters must be shared. Leave empty for single-instance / BYOK-only.
163181
- `IP_HASH_SALT` (required when `REDIS_URL` is set): salts the SHA-256 IP hash so persisted snapshots aren't brute-forceable. Generate with `openssl rand -hex 32`.
182+
- `DEMO_STT_OPENAI_API_KEY` (optional secret): an OpenAI key for the **demo** voice path only (`gpt-4o-transcribe` via `/api/transcribe`). Never the chat key, never a BYOK key. It is part of demo mode (see the combined bullet above): without it the deployment is **not** in demo mode, so demo chat **and** voice are both off and every visitor falls back to BYOK (which is browser-direct and needs no server key).
164183

165184
Once deployed, copy the `.ondigitalocean.app` URL DigitalOcean assigns and add it to your SimplePDF dashboard's whitelist before opening the app.
166185

@@ -221,6 +240,7 @@ The architecture is deliberate:
221240
- **Document data stays in the browser.** SimplePDF processes PDFs client-side. The iframe never uploads document bytes to SimplePDF.
222241
- **Chat traffic flows through your server.** You control the provider, the keys, the logs, and any RAG / internal data layered in.
223242
- **Submission is direct to your storage.** On Premium with [Bring Your Own Storage](https://simplepdf.com/pricing) (S3, Azure Blob, or SharePoint), completed PDFs upload from the browser to your bucket, never to SimplePDF servers.
243+
- **Voice input is the one exception, and it is opt-in.** Dictation sends the recorded audio clip out of the browser — unlike PDF bytes, dictated audio (which can contain PII/PHI) leaves the device. Two routes, each disclosed before recording: the **demo** route uploads to SimplePDF's server and on to OpenAI (the server keeps no audio and logs no transcript text — only an IP hash, byte size, and elapsed time); the **BYOK** route sends audio **directly to the user's chosen provider, never to SimplePDF** (SimplePDF makes no retention claim for user-selected providers — that's the provider's policy). Audio is sent only on an explicit Record.
224244

225245
### Using the demo account
226246

copilot/package-lock.json

Lines changed: 11 additions & 40 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

copilot/package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@
6161
"@types/react": "^19.2.0",
6262
"@types/react-dom": "^19.2.0",
6363
"@vitejs/plugin-react": "^6.0.1",
64+
"fake-indexeddb": "^6.2.5",
6465
"jsdom": "^28.1.0",
6566
"typescript": "^5.7.2",
6667
"vite": "^8.0.0",

0 commit comments

Comments
 (0)