fix: parse gpt-oss special token format in LM Studio responses #6740
Conversation
- Add detection for gpt-oss models in LmStudioHandler - Implement parseGptOssFormat method to extract actual content from special token format - Add comprehensive tests for the new parsing logic - Fixes #6739
dosubot
bot
added
size:L
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
| private parseGptOssFormat(content: string): string { | ||
| // Remove all special tokens and extract the actual message | ||
| // Pattern: <|token|> where token can be any word | ||
| const specialTokenPattern = /<\|[^|]+\|>/g |
There was a problem hiding this comment.
The regex pattern /<\|[^|]+\|>/g might not handle edge cases correctly. What happens if the token content itself contains a pipe character? Consider using a more robust parsing approach or documenting this limitation.
| for (const processedChunk of matcher.update(delta.content)) { | ||
| yield processedChunk | ||
| // Check if this is a gpt-oss model with special token format | ||
| const isGptOss = this.getModel().id?.toLowerCase().includes("gpt-oss") |
There was a problem hiding this comment.
Performance consideration: Since the model doesn't change during streaming, could we move this gpt-oss check outside the loop to avoid repeated string operations on every chunk?
| const isGptOss = this.getModel().id?.toLowerCase().includes("gpt-oss") | |
| // Check if this is a gpt-oss model with special token format | |
| const isGptOss = this.getModel().id?.toLowerCase().includes("gpt-oss") | |
| for await (const chunk of results) { | |
| const delta = chunk.choices[0]?.delta | |
| if (delta?.content) { | |
| if (isGptOss && delta.content.includes("<|") && delta.content.includes("|>")) { |
| // Parse gpt-oss special token format | ||
| // Format: <|start|>assistant<|channel|>commentary to=read_file <|constrain|>json<|message|>{"args":[...]} | ||
| const cleanedContent = this.parseGptOssFormat(delta.content) | ||
| if (cleanedContent) { |
There was a problem hiding this comment.
When cleanedContent is empty or falsy after parsing, we silently skip it. Should we consider logging a warning to help with debugging unexpected formats?
| }) | ||
| }) | ||
|
|
||
| describe("gpt-oss special token parsing", () => { |
There was a problem hiding this comment.
Consider adding test cases for edge scenarios:
- Multiple
<|message|>tokens in a single chunk - Malformed special tokens (e.g., unclosed tokens like
<|start) - Very large JSON payloads after the message token
- Mixed content with both special tokens and regular text
| const cleaned = content.replace(specialTokenPattern, " ").trim() | ||
|
|
||
| // Also clean up any "to=function_name" patterns that might remain | ||
| const functionPattern = /\s*to=\w+\s*/g |
There was a problem hiding this comment.
The function pattern /\s*to=\w+\s*/g only matches word characters. Is this intentional? Function names might contain hyphens or underscores. Consider using /\s*to=[\w-]+\s*/g if you want to support kebab-case function names.
hannesrudolph
added
the
Issue/PR - Triage
|
The model is hallucinating a format and looping, closing for now |
This PR fixes the issue where gpt-oss models output responses in a special token format that was being displayed raw instead of being parsed properly.
Problem
When using LM Studio with gpt-oss-20b model, responses were appearing in this format:
Solution
parseGptOssFormatmethod that:<|message|>token if presentTesting
Fixes #6739
Important
Fixes gpt-oss model response parsing in LM Studio by adding special token format handling in
LmStudioHandler.LmStudioHandlernow detects gpt-oss models and parses special token formats usingparseGptOssFormat().<|message|>token or removes special tokens and function patterns if not present.lmstudio.spec.tsfor gpt-oss format parsing with and without<|message|>token.This description was created by
for 93dc32d. You can customize this summary. It will automatically update as commits are pushed.