-
Notifications
You must be signed in to change notification settings - Fork 2.5k
feat: add thinking/reasoning data support to LiteLLM provider #8498
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
- Handle reasoning, thinking, and reasoning_content fields in streaming responses - Pass through thinking data from underlying models as reasoning chunks - Add comprehensive test coverage for all thinking data scenarios - Fix handling of chunks without choices array Fixes #8497
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Self-review: judging my own code like a mirror wired to unit tests—existential and merciless.
| // Check for reasoning/thinking content in the delta | ||
| // LiteLLM may pass through reasoning content from underlying models | ||
| if (delta) { | ||
| if ("reasoning" in delta && delta.reasoning && typeof delta.reasoning === "string") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Emit at most one reasoning chunk per delta to avoid duplicated output when multiple alias fields are present in a single chunk.
| if ("reasoning" in delta && delta.reasoning && typeof delta.reasoning === "string") { | |
| if ("reasoning" in delta && delta.reasoning && typeof delta.reasoning === "string") { | |
| yield { type: "reasoning", text: delta.reasoning } | |
| } else if ("thinking" in delta && delta.thinking && typeof delta.thinking === "string") { | |
| yield { type: "reasoning", text: delta.thinking } | |
| } else if ( | |
| "reasoning_content" in delta && | |
| delta.reasoning_content && | |
| typeof delta.reasoning_content === "string" | |
| ) { | |
| yield { type: "reasoning", text: delta.reasoning_content } | |
| } |
| yield { type: "reasoning", text: delta.reasoning_content } | ||
| } | ||
|
|
||
| if (delta.content) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Add a type guard for delta.content to ensure only string fragments are emitted (prevents accidental non-string values from being yielded).
| if (delta.content) { | |
| if (typeof delta.content === "string" && delta.content) { | |
| yield { type: "text", text: delta.content } | |
| } |
Review in progress. Flagged two P2 polish items to harden LiteLLM streaming handling.
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
| if ("reasoning" in delta && delta.reasoning && typeof delta.reasoning === "string") { | ||
| yield { type: "reasoning", text: delta.reasoning } | ||
| } | ||
|
|
||
| // Also check for thinking content (alternative field name) | ||
| if ("thinking" in delta && delta.thinking && typeof delta.thinking === "string") { | ||
| yield { type: "reasoning", text: delta.thinking } | ||
| } | ||
|
|
||
| // Check for reasoning_content (another possible field name) | ||
| if ( | ||
| "reasoning_content" in delta && | ||
| delta.reasoning_content && | ||
| typeof delta.reasoning_content === "string" | ||
| ) { | ||
| yield { type: "reasoning", text: delta.reasoning_content } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Emit at most one reasoning chunk per delta to avoid duplicate output when providers send multiple alias fields in a single delta (e.g., both reasoning and thinking). Using an else-if chain preserves first-match precedence and prevents duplicates.
| if ("reasoning" in delta && delta.reasoning && typeof delta.reasoning === "string") { | |
| yield { type: "reasoning", text: delta.reasoning } | |
| } | |
| // Also check for thinking content (alternative field name) | |
| if ("thinking" in delta && delta.thinking && typeof delta.thinking === "string") { | |
| yield { type: "reasoning", text: delta.thinking } | |
| } | |
| // Check for reasoning_content (another possible field name) | |
| if ( | |
| "reasoning_content" in delta && | |
| delta.reasoning_content && | |
| typeof delta.reasoning_content === "string" | |
| ) { | |
| yield { type: "reasoning", text: delta.reasoning_content } | |
| } | |
| if ("reasoning" in delta && typeof delta.reasoning === "string" && delta.reasoning) { | |
| yield { type: "reasoning", text: delta.reasoning } | |
| } else if ("thinking" in delta && typeof delta.thinking === "string" && delta.thinking) { | |
| yield { type: "reasoning", text: delta.thinking } | |
| } else if ( | |
| "reasoning_content" in delta && | |
| typeof delta.reasoning_content === "string" && | |
| delta.reasoning_content | |
| ) { | |
| yield { type: "reasoning", text: delta.reasoning_content } | |
| } |
Fix it with Roo Code or mention @roomote and request a fix.
| if (delta.content) { | ||
| yield { type: "text", text: delta.content } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a type guard for delta.content so only string fragments are emitted. Some providers may send non-string types, which would surface as [object Object] or cause UI issues.
| if (delta.content) { | |
| yield { type: "text", text: delta.content } | |
| } | |
| if (typeof delta.content === "string" && delta.content) { | |
| yield { type: "text", text: delta.content } | |
| } |
Fix it with Roo Code or mention @roomote and request a fix.
Summary
This PR attempts to address Issue #8497 by adding support for displaying thinking/reasoning data from LiteLLM in Roo Code's thinking block.
Problem
LiteLLM can display thinking data from underlying models, but Roo Code wasn't capturing and displaying this data in the thinking block.
Solution
reasoning,thinking,reasoning_content) for compatibility with different LiteLLM model responsesApiStreamReasoningChunkobjects to be displayed in the UIChanges
src/api/providers/lite-llm.tsto handle thinking/reasoning fields in streaming responsessrc/api/providers/__tests__/lite-llm.spec.tsTesting
reasoningfield in deltathinkingfield in deltareasoning_contentfield in deltaReview Confidence
Implementation review showed 92% confidence with no issues identified.
Fixes #8497
Feedback and guidance are welcome!
Important
Adds support for handling and displaying thinking/reasoning data from LiteLLM in Roo Code's UI.
LiteLLMHandlerinlite-llm.tsnow detectsreasoning,thinking, andreasoning_contentfields in streaming responses, yielding them asApiStreamReasoningChunkobjects.lite-llm.spec.tsfor handlingreasoning,thinking, andreasoning_contentfields.lite-llm.ts.This description was created by
for dab8d4c. You can customize this summary. It will automatically update as commits are pushed.