Try streaming responses (research + prototype)

[dokuczacz]

Goal: evaluate and (if feasible) implement streaming for chat responses.

Notes:

• Requires analysis of streamz repo: https://github.com/python-streamz/streamz.git
• Decide streaming approach (SSE/WebSocket/chunked fetch) across ui_next + backend.

Acceptance:

• Prototype streaming in UI with a simple backend stream mode
• Document constraints/rollback (feature flag)

[dokuczacz]

Update (Feb 2026): Tool handler refactor completed via PR #21

Current State:

• ✅ WP6 FAST mode optimized for low-latency responses (Phase 4)
• ✅ Prompt caching reduces initial latency by 80-90%
• ✅ Context builder has AUTO/FAST/DEEP routing with intelligent mode selection
• ✅ Full documentation in PROMPT_CACHING_GUIDE.md

Performance Improvements:

• FAST mode: Minimal context, <2s typical response time
• Prompt caching: 80-90% cache hits reduce latency significantly
• Cost reduction: 28-40% savings from caching optimization

Streaming Considerations:

• Current architecture supports tool-by-tool execution
• FAST mode may already meet "fast perception" requirements
• True streaming could be added at execution level if still needed

Recommendation:

1. First: Test FAST mode performance in production with prompt caching enabled
2. Then: If latency still feels slow, evaluate streaming implementation
3. Consider: Streaming may be most valuable for DEEP mode (long responses)

Next Steps:

• Deploy with prompt caching enabled
• Measure actual user perception of response times
• Determine if streaming is still needed given current optimizations

Keeping open pending real-world performance evaluation. May close if FAST+caching meets UX goals.