New endpoints (reasoning for the exising ml cmds & copilot) (#1322)

* feat(ml): propagate conversation_id across KCL flows, keep receipts

- Add conversation_id to edit requests and responses for session tracking
- Update kittycad to 0.3.39, tidy Cargo.toml feature lists

Signed-off-by: Jessie Frazelle <[email protected]>

* chore(deps): bump kittycad to 0.3.40, because numbers go up

- Picks up latest fixes, keeps clap, tabled, requests features
- No code changes, just a dependency bump

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(cli): stream reasoning via WS, add --no-reasoning (no spoilers)

- Show live model reasoning during long runs via reasoning_ws over
  tokio-tungstenite, printed to stderr with concise colored output
- Enabled by default, add --no-reasoning to mute; docs and tests
  updated

Signed-off-by: Jessie Frazelle <[email protected]>

* test(ml): reasoning streams by default, --no-reasoning hits mute

- Add integration tests for text-to-cad export reasoning to stderr,
  on by default, muted with --no-reasoning
- Drop brittle docs assertion, tidy imports and formatting

Signed-off-by: Jessie Frazelle <[email protected]>

* test(kcl): e2e for ml kcl edit in temp dir, no repo mess

- Use a temp project, copy tests/gear.kcl, keep dir alive in vec to avoid
  touching repo files
- Add e2e for ml kcl edit with and without reasoning; assert write and 0 exit

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(cli): unify output format parsing, no more enum dupes

- Custom parser: 'kcl' handled inline, others use kittycad enums with
  clearer errors
- Simplified model selection; removed local enum and TryFrom boilerplate

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(copilot): don't panic on errors, bail on EOS

- handle server Error messages with red stderr, keep stream alive
- stop on EndOfStream and close frames; close ws cleanly
- silence connect failures to avoid noisy debug spew

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(context): RAII guard for reasoning WS, no more zombie tasks

- Wrap reasoning WS task in RAII ReasoningGuard, abort and await
  safely; replace ad hoc aborts with guard.finish
- Prevent leaked tasks on early returns and timeouts; minor fmt tidy
  for eprintln with {line} and reflowed serde_json parse

Signed-off-by: Jessie Frazelle <[email protected]>

* Update src/context.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update src/context.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* feat(kcl): add Copilot chat CLI, finally a coworker that responds

- Introduce 'zoo ml kcl copilot' REPL that streams replies and prints
  reasoning, tool output, and errors
- Snapshot cwd files for context, support --project-name, make
  print_reasoning pub(crate)

Signed-off-by: Jessie Frazelle <[email protected]>

* chore(deps): bump kittycad to 0.3.41, thrilling I know

- Pull in upstream fixes, keep deps fresh

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(context): abort once then await in bg; kill zombie tasks, Morty

- Replace duplicate abort with background await to drive completion
- Keep drop non-blocking and avoid leaked join handles

Signed-off-by: Jessie Frazelle <[email protected]>

* test(cli): no more vibes, assert "reasoning:" and gear.kcl

- stderr: expect explicit "reasoning:" label, not any colon
- stdout: match filename "gear.kcl" instead of "Wrote to", less brittle

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(ml): ratatui Copilot chat, no more stdin karaoke

- Replace REPL with ratatui chat UI with streaming and reasoning
- Keys: Enter sends, Shift+Enter newline, Ctrl+C exits
- Factor copilot into ml::copilot; add crossterm and ratatui

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot,tui): async file scan with progress, so it stops freezing

- Scan files in the background with progress (spawn_blocking + chan),
  attach files when ready, and drop the old gather_cwd_files. No more
  startup freeze, you're welcome, Jerry
- Gate user sends with single flight and a queue, release on
  EndOfStream or after scan; TUI shows Scanning files and Waiting for
  response; tests and debug logs included

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(ml/copilot): main.kcl or bust, scan only relevant files

- Validate project before launch, require main.kcl via existing
  discovery for consistent errors
- Scan only kcl_lib::RELEVANT_FILE_EXTENSIONS; less I/O; add test for
  missing main.kcl

Signed-off-by: Jessie Frazelle <[email protected]>

* docs(kcl): Copilot chat needs main.kcl in current dir, who knew

- Clarify requirement so users know chat fails without a main.kcl

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(copilot): extract scan_relevant_files, add tests, less spam

- Extract scan to scan_relevant_files, keep extension allowlist,
  skip .git, target, node_modules, hidden; unit test added
- Progress now only sends a final count to cut TUI noise; misc fmt tidy

Signed-off-by: Jessie Frazelle <[email protected]>

* updates

Signed-off-by: Jessie Frazelle <[email protected]>

* style(tests): use captured fmt in panic, because 2025 exists

- Use captured identifiers in panic formatting for clarity and
  consistency
- No behavior change

Signed-off-by: Jessie Frazelle <[email protected]>

* style(tests): reorder imports to appease rustfmt, happy now Jerry

Signed-off-by: Jessie Frazelle <[email protected]>

* updates

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(copilot-tui): shush logs in TUI, better ws debug, flush, Morty

- Mute global logs in TUI, restore on exit; debug stays in chat
- Harden WS diagnostics: log close codes and errors, add connect msg;
  flush after send, drop keypress spam

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot): shrink payloads, first-msg, WS ping (size matters)

- Env knobs: ZOO_COPILOT_ATTACH (all, main, none) selects files, and
  ZOO_COPILOT_MAX_JSON caps JSON. Attach only on first message, we
  auto-shrink to main or none if needed
- Dedicated WS writer with ping keepalive, graceful close, better
  debug, plus auth preflight before launching the TUI

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot): debug payload logs, chunked and capped, no eye bleed

- Add optional payload logging when ctx.debug, controlled by
  ZOO_COPILOT_LOG_PAYLOAD
- Chunk output into ~1k lines and truncate via ZOO_COPILOT_LOG_LIMIT
  (default 10k) to keep the TUI readable

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(kcl): choose your Zoo host for Copilot (-H/--host, ZOO_HOST), Morty

- Add -H/--host and ZOO_HOST to pick Zoo host, https by default
- Route auth and calls to the chosen host in the TUI

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(copilot): stop gatekeeping TUI behind auth, let it boot offline

- Remove preflight users().get_self() auth check
- Defer auth to first TUI API call, enabling offline startup and avoiding a redundant network call

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(run)!: RIP env knobs, hardcode limits, attach All first

- Hardcode limits: MAX_JSON_LEN=50k, PAYLOAD_LOG_LIMIT=10k
- Attach All first, then shrink; log payload only with ctx.debug

BREAKING CHANGE: ZOO_COPILOT_ATTACH, ZOO_COPILOT_MAX_JSON,
ZOO_COPILOT_LOG_PAYLOAD, and ZOO_COPILOT_LOG_LIMIT are no longer used

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(ui): ML-ephant> everywhere, dim reasoning, flush on meta (finally)

- Unify prefixes: ML-ephant> bot, You> user; info/error/tool use it
- Dim reasoning; flush buffered replies before meta for sane order

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(copilot/ws): type-safe writer, no more raw Message soup

- Add WsSend enum for outbound WS traffic (Client, Ping, Close), so the
  writer is type-safe and not juggling raw Message
- Centralize JSON serialization and payload logging in the writer,
  cutting duplicate UI logs; introduce emit_payload_lines and
  build_user_message_with_fallback (typed, returns len, shrink kept)

Signed-off-by: Jessie Frazelle <[email protected]>

* perf(copilot): batch info/reasoning messages, TUI stops screaming

- Introduce ServerMsgBatcher to accumulate info and plain reasoning
  text, trim trailing newline on flush, flush on structured messages
  and at end
- Replace dbg closure with batcher, cut channel chatter and debug spam,
  fewer TUI redraws

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot): diffs from tool outputs, /accept or /reject (ask first)

- Handle ToolOutput, build unified diffs via similar, queue as
  pending edits, preview in TUI; add tests
- Add /accept and /reject to apply or discard edits to disk

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot): Tab completes slash commands, because life is short
- Autocomplete unique, extend common prefix, else Info suggestions
- Preserve Tab indent outside slash, tests cover completion and hints

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot): add scrolling for chat and diff, my eyes say thanks

- Handle PgUp, PgDn, Up, Down to scroll chat or diff
- Add msg_scroll and diff_scroll; clamp diff offset; add tests

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(ui): wrap chat messages; verify scroll, no more spill

- Wrap chat messages to prevent overflow, keep spacing intact
- Add tests for message wrapping and scroll; ensure diff header scrolls

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(copilot-tui): kill inline /accept, /reject; one parser to rule

- Remove local handlers for /accept and /reject in run_copilot_tui
- Centralize command handling in app.try_submit to avoid duplication

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(copilot): RIP AttachMode, no batching, always attach files

- Always attach all files, remove AttachMode and shrinking
- Drop reader batching, forward server messages immediately

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(copilot): drop _send_every_time and sent_files_once (bye noise)

- Remove _send_every_time from build_user_message, update calls and tests
- Behavior unchanged, we always sent files when present

Signed-off-by: Jessie Frazelle <[email protected]>

* style(run): drop stale test comment left after refactor, truly riveting

Signed-off-by: Jessie Frazelle <[email protected]>

* test(copilot): exorcise carryover, no ghosts between turns

- Lock down no message carryover across client, event loop, and UI
- Verify EOS clears awaiting state and files are sent each turn

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot): WS content preview, reasoning gets one line, not a novel

- Outgoing WS debug shows content preview and current_files keys,
  trimmed and capped for readable logs
- Reasoning renders as a single line, show_reasoning removed; Info,
  Error, and ToolOutput print as standalone lines

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(ui): Info learns Markdown, multiline output stops shouting

- Parse Info with pulldown_cmark, emit lines for headings, lists, code
- Show multi-line Reasoning, Error, ToolOutput line by line; add test

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(copilot-ui): render diff or chat, not both (no UI Jenga)

- Render chat only when no pending edits, diff view is exclusive
- Frame diff with Chat block for consistency, add test to prevent overlay

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(copilot-ui): drop redundant enumerate, label never changed, duh

- Remove enumerate and dead branch in error and tool output
- No behavior change

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot-ui): render reasoning as Markdown, dimmed for human eyes

- Add context::reasoning_to_markdown, outputs Markdown with bold
  headers, fenced KCL, and pretty JSON so the Copilot pane is not a
  wall of text
- UI now renders markdown to lines and dims reasoning text, replaces
  format_reasoning for cleaner output

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot-ui): Markdown renders like a grown-up, fences + live

- Robust Markdown parsing: headings, lists, code fences with language,
  preserved line breaks and horizontal rules
- Live-render deltas with per-line ML-ephant prefix; add test; remove
  noisy diff pane border

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(copilot/ui): bye code_lang, hello simpler fenced blocks

- Simplify code fence rendering, inline match on kind, emit lang label
- Less state and fewer allocs, same output; plus minor style cleanup

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(ui): stream raw lines, preserve newlines, markdown can chill

- Swap to render_preserving_newlines for assistant streaming and
  EOS, avoid parsing half-baked markdown mid-stream
- Preserve blank lines and raw markers; update test to expect
  '# Title' and list dashes

Signed-off-by: Jessie Frazelle <[email protected]>

* style(copilot-ui): silence ML-ephant>, use 4-space gutter

- Replace assistant prefix with 4-space indent, less noise, more signal
- Update diff rendering and tests to match, keep You> and colors intact

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot-ui): no more prefix spam, single green assistant label

- Introduce ASSISTANT_LABEL, push_assistant_block, preserve newlines
- Unifies reasoning, info, errors, tool output, first-line prefix only

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot): add system slash commands, autocomplete; tab me, Morty

- Wire in SDK system commands as /cmd, tab-complete and hints; auto-syncs
- Send System over WS; SlashCommand now carries data, tests added

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(cli): add global --host, repeating flags is for Jerrys

- Add global --host (env ZOO_HOST) via Context override; host resolution is
  explicit arg, global override, then config default
- Remove per-command host flags from auth login/logout/status and kcl; update
  help, examples, tests, and logout non-interactive error text

BREAKING CHANGE: per-command -H/--host options removed. Use
zoo --host <host> <command> ... instead

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot): /render side-by-side snapshots, eyes > diff hunks

- Add /render to compare pending edits visually, stitch old and new PNGs
- Respect project.toml executor settings; add image and tempfile deps

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(tui): preview rendered diff in terminal, because eyes > logs

- Show rendered comparison via viuer, wait for key, then clear screen
- If preview fails, emit error event and still report saved path

Signed-off-by: Jessie Frazelle <[email protected]>

* feat(copilot): add /render preview to diff header, look before commit

- Mention /render in diff header to preview before applying
- Add tests for /render in UI buffer and server info message

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(copilot): block path traversal with secure joins (no portals)

- Guard all file writes and previews with join_secure, reject absolute
  and .. paths; normalize diff headers to project-relative paths.
- Bubble IO errors to the UI, inform on new files, abort bad previews;
  add tests; show "Press any key to return..." hint in image preview.

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(copilot): ditch recursion in file scan, no more stack faceplants

- Iterative dir scan to avoid stack overflow on deep trees
- Add deep nesting test so we do not faceplant again

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(copilot): stop following symlinks, we’re not a cat
- Skip symlink targets during file scan to prevent cycles and hangs
- Add tests for path traversal rejection, host resolution, and /render

Signed-off-by: Jessie Frazelle <[email protected]>

* fix(copilot): lock down path join and scan, no portal gun escapes

- Harden paths: canonicalize base, normalize, block traversal and
  symlink escapes; scanner adds depth cap (256), skips symlinks,
  TOCTOU guard, tests included.
- Simplify reasoning markdown, return raw text for Text variant

Signed-off-by: Jessie Frazelle <[email protected]>

* refactor(copilot): extract path utils, fewer symlink oops Morty

- Move join_secure and scan_relevant_files to util, add tests for
  symlink escapes and depth; reuse from run.rs
- Add test: reasoning_to_markdown(Text) renders without a header

Signed-off-by: Jessie Frazelle <[email protected]>

---------

Signed-off-by: Jessie Frazelle <[email protected]>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

Author: jessfraz