Skip to content

feat: token-level streaming for pack SSE routes #86

Description

@Brescou

Context

Pack SSE streams (/packs/{id}/run/stream and the legacy pipeline stream) currently emit phase-level events (phase_started, phase_completed, pipeline_completed). Clients see nothing between phase boundaries — no incremental output while the LLM generates.

Token-level streaming (each LLM token forwarded as it is generated) is the UX baseline users expect today, and a prerequisite for AG-UI support (#planned).

Scope

  • Propagate LangGraph token streaming (stream_mode="messages" or equivalent) through agent nodes up to the SSE endpoints, alongside the existing phase events (keep those — they are useful for progress UI).
  • Decide the event schema: e.g. {"type": "token", "phase": "analysis", "content": "..."}.
  • CostTracker interaction: token accounting on an in-flight stream, and what happens when the budget is exhausted mid-stream (emit an explicit error event, consistent with the existing SSE error behaviour).
  • Structured-output packs (StructuredLLMPack) don't stream naturally (validated JSON) — token streaming may apply only to free-text phases, with structured packs keeping phase-level events. To be settled at design time.

Non-goals

  • No UI work; this is transport-level only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions