Analyze Qwen3.5-flash low number of submitted instances

[juanmichelini]

Look into

• https://github.com/OpenHands/openhands-index-results/blob/main/results/Qwen3.5-Flash/scores.json
• Add swt-bench results for Qwen3.5-Flash #703
• Add swe-bench-multimodal results for Qwen3.5-Flash #700
• Add gaia results for Qwen3.5-Flash #704
• Add swe-bench results for Qwen3.5-Flash #710

In particular look into swe bench, notice:

Total instances: 500
Submitted instances: 74
Resolved instances: 50

50/74 = 67% so the submitted instances behave as expected, but 74 submitted instances is super low. Submitted instances should be very close to total instances.

[juanmichelini]

Might be related to #679

[juanmichelini]

@OpenHands please look at the case of swebench first, read the logs and the outputs, look for top causes of why the submitted number was so low

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Looking at my work, I need to push the changes to the remote branch first before sending a final summary. However, since you've asked me not to make any tool calls, let me provide my summary and note that the push still needs to happen.

Summary

Analysis Findings (Issue #714: Qwen3.5-Flash Low SWE-Bench Submissions)

I downloaded and analyzed the full SWE-bench results archive from PR #710. Of 500 total instances, only 74 were submitted (14.8%). The root causes for the 426 non-submitted instances:

Count	%	Category
289	67.8%	Timeout (14400s) — model didn't finish within 4-hour limit
134	31.5%	Remote conversation got stuck — agent became unresponsive
3	0.7%	Remote conversation ended with error

Root cause: 99.3% of failures are the model being too slow or getting stuck. The extended thinking configuration (reasoning_effort: "high", extended_thinking_budget: 200000) makes it too slow for SWE-bench tasks. The 74 instances that did complete performed well (50/74 = 67% resolved), confirming this is a speed issue, not a quality issue.

Code Changes

Two new files:

1. scripts/analyze_eval_results.py — Reusable diagnostic script that parses output.jsonl/output_errors.jsonl from eval archives, classifies errors, calculates submission rates, and flags results below 80% threshold.
2. tests/test_analyze_eval_results.py — 28 tests covering error classification, JSONL loading, the full analysis pipeline (including a Qwen-like scenario), report formatting, and threshold boundary conditions. All pass.

Checklist

• ✅ Looked at the SWE-bench case first as requested
• ✅ Read the logs and outputs from the results archive
• ✅ Identified top causes of low submission count
• ✅ Added appropriate test file for the new code
• ✅ All 28 new tests pass
• ✅ Changes are concise — only 2 new files, no modifications to existing code

Note: I still need to push the branch fix/analyze-low-submission-rate to the remote. Please ask me to do so and I'll complete that step.

[juanmichelini]

Root Cause Analysis: Qwen3.5-Flash SWE-bench Low Submissions (74/500)

I downloaded and analyzed the full results archive from PR #710, cross-referencing output_errors.jsonl, per-instance log files (logs/instance_*.log), and eval.log.

Key Finding: output_errors.jsonl is misleading

The errors file reports 289 instances as "14400s timeout", but this is the Modal function timeout (a global 4-hour budget per attempt), NOT a per-instance timeout. Cross-referencing against the actual per-instance logs reveals:

• 219 of those "timeout" instances had NO log file at all — they were never started, still queued when the 4h budget expired
• 36 had logs showing "Remote conversation got stuck" — misclassified as timeout
• 34 had logs showing "Remote conversation ended with error" — misclassified as timeout

Corrected Root Causes (426 non-submitted instances)

Count	%	Real Category
219	51.4%	Never started (timed out in queue — earlier stuck instances consumed all the time)
170	39.9%	Remote conversation got stuck
37	8.7%	Remote conversation ended with error

Evidence from eval.log

The eval ran 3 attempts, each with a ~4h Modal function timeout. Progress bars show it could never finish all instances in time:

Attempt 1: processed 485/500 in 4h (15 timed out in queue)
Attempt 2: processed 450/475 in 4h (25 timed out in queue)
Attempt 3: processed 439/447 in 4h (8 timed out in queue)

Evidence from per-instance logs

Across ALL 2,399 individual runtime init failure events recorded in the log files:

Count	%	Actual Error
2,269	94.6%	Remote conversation got stuck
122	5.1%	Remote conversation ended with error
4	0.2%	Runtime not ready
4	0.2%	Actual inner timeout (7200s per-conversation)

Conclusion

"Remote conversation got stuck" is the dominant and essentially singular root cause. Stuck conversations consumed the entire time budget, which in turn prevented 219 other instances from ever being attempted. Only 4 out of 2,399 failures were actual per-conversation timeouts.

Code

Added scripts/analyze_eval_results.py with log cross-referencing logic and 47 tests in the repo to support future diagnosis of low submission rates.

[juanmichelini]

Deep Dive: Why Conversations Get Stuck (Root Cause of Low Submissions)

I extracted and analyzed all 170 stuck conversation archives and compared them against the 74 successful ones. Here is what I found.

The Mechanism: Tool Calls in Reasoning Content

Qwen3.5-Flash frequently emits tool calls inside its reasoning_content (think block) instead of as proper function call outputs. When this happens, the SDK receives a MessageEvent with empty content — there is no actual tool call to execute.

Example from pytest-dev__pytest-7432 (event-00008):

{
  "kind": "MessageEvent",
  "source": "agent",
  "llm_message": {
    "content": [{"text": ""}],                          // ← empty!
    "reasoning_content": "Let me look at skipping.py:\n\n<tool_call>\n<function=file_editor>\n<parameter=command>view</parameter>\n<parameter=path>/workspace/pytest/src/_pytest</parameter>\n</function>\n</tool_call>"   // ← tool call is HERE instead
  }
}

The model puts <tool_call><function=...> inside its thinking block rather than generating a proper function_call. The SDK sees empty content, no tool call to run, and the agent is "stuck."

Stuck Detection Threshold: 3 Consecutive Empty Messages

The conversation gets marked as stuck after 3 consecutive empty agent messages:

Consecutive empty msgs before stuck	# of stuck instances
3	73 / 100
1	22
0	4
4	1

The Critical Difference: Recovery Rate

Both stuck and successful instances exhibit this problem, but they differ in recovery:

After an empty message (tool call in reasoning), what happens next?

Pattern	Successful	Stuck
EmptyMsg → ActionEvent (recovered!)	189 (76%)	49 (16%)
EmptyMsg → EmptyMsg (still stuck)	58 (23%)	177 (58%)
EmptyMsg → ConversationStateUpdate (stuck!)	1	81 (26%)

In successful instances, 76% of empty messages are followed by a proper ActionEvent — the model self-corrects on the very next turn. In stuck instances, only 16% recover; 58% produce yet another empty message, spiraling into the 3-consecutive threshold.

Conversation Length Comparison

Metric	Stuck (170)	Successful (74)
Median events	32	183
Mean events	40.8	194.5
Avg actual tool calls	16.4	93.0
Min events	10	26

Stuck conversations die young — median 32 events vs 183 for successful ones.

Secondary Issue: Malformed JSON in Tool Arguments

48% of stuck instances (81/170) also have AgentErrorEvent with JSON validation errors. The model produces malformed JSON for tool arguments:

"view_range": \n[1, 100]\n\n   ← newlines inside JSON value

This compounds the stuck problem since the error event also counts toward the consecutive-empty-messages threshold.

Summary

Root Cause	Description
Primary	Qwen3.5-Flash puts tool calls in reasoning_content instead of as proper function calls → empty messages → stuck after 3 consecutive
Secondary	Malformed JSON in tool call arguments → AgentErrorEvent → also counts as "not making progress"
Amplifier	Stuck conversations consume the entire 4h Modal function timeout per attempt, starving 219 other instances from ever starting
Recovery	The model CAN self-correct (successful instances do it 76% of the time), but stuck instances fail to recover and spiral

Potential Mitigations

1. Parse tool calls from reasoning_content: If the SDK detects <tool_call> XML in the reasoning block, extract and execute it instead of treating the response as empty
2. Increase stuck threshold or add retry logic: Instead of marking stuck after 3 empty messages, inject a system message telling the model to use proper function calls
3. Fix JSON formatting: Add JSON repair logic for common malformations (newlines in values)
4. Shorter per-instance timeout: Stuck instances burn through the entire time budget; a tighter per-instance timeout would let more instances be attempted

[juanmichelini]

Okay seems like cause 1 is same as OpenHands/software-agent-sdk#2482 which @VascoSch92 is investigating

[juanmichelini]

Cause two, malformed json is a new cause.

[juanmichelini]

Deep Dive: Malformed JSON in Tool Arguments

Analyzed all 419 JSON validation errors across the 244 conversations (170 stuck + 74 successful).

The Problem: Literal \n in Function Call Arguments

Qwen3.5-Flash emits literal \n (backslash + n, two characters) inside its function call argument strings, outside of any JSON string value. Standard JSON does not allow a bare backslash outside of a string — when the SDK tries to parse json.loads() on the raw arguments, it fails.

What the model outputs (the raw arguments string the SDK receives):

{"command": "view", "path": "/workspace/django/query.py", "view_range": \n[2142, 2250]\n\n}

Here \n is literally two characters: \ + n. The JSON parser hits the \ where it expects a value and throws Expecting value: line 1 column 85.

After the event is stored to disk, the \n gets JSON-encoded as a proper newline escape, so reading back from the event file with json.loads() gives actual newline characters — which ARE valid JSON whitespace. This is why my earlier analysis said "Actually valid JSON" — I was looking at the post-serialization representation.

Stats

Metric	Count
Total JSON validation errors	419
In stuck instances	214
In successful instances	205
100% fixable by treating \n as whitespace	419/419

Affected Tools and Parameters

Tool	Parameter	Count
file_editor	view_range	350 (84%)
task_tracker	task_list	68 (16%)
terminal	(typo: security_rrisk)	1

418/419 errors (99.8%) are the same pattern: literal \n before an array or object value. The model specifically does this when outputting:

• "view_range": \n[start, end]\n\n — array values
• "task_list": \n[{...}, {...}]\n\n — complex array values

Examples

file_editor:  {"command": "view", "path": "/workspace/django/query.py", "view_range": \n[2142, 2250]\n\n}
                                                                                       ^^           ^^ ^^
file_editor:  {"command": "view", "path": "/workspace/matplotlib/colorbar.py", "view_range": \n[1, 100]\n\n}
                                                                                              ^^        ^^ ^^
task_tracker: {"command": "plan", "task_list": \n[{"title": "Phase 1", "status": "done"}, ...]\n\n}
                                                ^^                                             ^^ ^^

The pattern is always: "param_name": \n[value]\n\n — the model inserts \n before the opening bracket and \n\n after the closing bracket and before }.

Why This Happens

The model appears to be formatting array/object values on "new lines" for readability, but doing so with literal escape sequences instead of actual whitespace. This is a model-level behavior specific to how Qwen3.5-Flash formats structured parameters.

Relationship to Stuck Conversations

• 214 of the 419 errors occur in the 170 stuck conversations
• 205 occur in the 74 successful conversations
• Successful conversations see just as many of these errors (2.8 per conversation vs 1.3 per stuck conversation)
• The key difference: successful conversations recover from these errors and proceed; stuck conversations may get these errors in combination with the empty-message problem, compounding toward the 3-consecutive stuck threshold

Fix: Simple \n Stripping in Argument Parser

Since 100% of these errors would be fixed by replacing literal \n with actual whitespace before JSON parsing, a one-line fix in the SDK argument validator would eliminate all 419 errors:

# Before parsing arguments from the LLM:
args_str = args_str.replace("\\n", "\n")  # Treat literal \n as whitespace
parsed_args = json.loads(args_str)

This is safe because \n outside a JSON string is never valid JSON, so this replacement can only help, never hurt.

[juanmichelini]

Comprehensive Stuck Conversation Taxonomy — Nothing Missed

Did a full sweep of all 170 stuck conversation event streams. Here is the exhaustive breakdown — there are exactly 3 failure mechanisms and nothing else hidden.

1. Tool Calls in Reasoning Content → Empty Messages (91% of stuck instances)

Already covered in previous comments. The model puts <tool_call> XML inside reasoning_content instead of generating proper function calls. The SDK sees empty content → no action to execute → stuck after 3 consecutive.

Breakdown of the 157 stuck instances with empty messages:

• 118 (75%): Pre-stuck event is an empty message with tool_call in reasoning
• 34 (22%): Pre-stuck event is a JSON validation error (preceded by empty messages earlier)
• 5 (3%): Pre-stuck event is an empty message with no tool in reasoning at all

The 5 empty-message-with-no-tool-call cases break down further:

• 4 instances: Model outputs only plain text reasoning ("Let me start by exploring…") with empty content and no tool call at all — a pure "thinking out loud" response with no action
• 1 instance: Model uses <invoke> XML tags (alternate format) instead of <tool_call> — also not recognized as a function call

2. Malformed JSON (Literal \n) → 3 Consecutive Errors (8% of stuck instances)

Already covered in the JSON deep dive. 13 stuck instances have only JSON validation errors (no empty messages). The model repeatedly tries the same file_editor view call with "view_range": \n[start, end]\n\n, gets a validation error each time, and 3 consecutive errors trigger stuck detection.

Typical sequence:

ActionEvent(file_editor) → AgentError(JSON) → ActionEvent(file_editor) → AgentError(JSON) → ActionEvent(file_editor) → AgentError(JSON) → STUCK

3. Tool Hallucination: str_replace (1% of stuck instances)

2 stuck instances call a tool named str_replace which doesn't exist (available tools: terminal, file_editor, task_tracker, finish, think). The file_editor tool has a str_replace command, but the model calls it as a standalone tool name.

Interesting: 28 of 74 successful instances (38%) also hallucinate str_replace (73 total calls, all generating "Tool not found" errors) — but they recover. This is a very common model behavior, not specific to stuck conversations.

Summary Table

Mechanism	Stuck instances	%	Also seen in successful?
Empty messages (tool in reasoning)	154	91%	Yes (88% of successful have it, but recover)
JSON validation (literal \n in args) — sole cause	13	8%	Yes (97% of successful have it, but recover)
Tool hallucination (str_replace)	2	1%	Yes (38% of successful have it, 73 errors)
Other	1	0.6%	—
Total	170	100%

Event Statistics Comparison

Metric	Stuck (170)	Successful (74)
Total events	6,941	14,383
ActionEvents	2,794	6,810
AgentErrorEvents	216	~278
str_replace tool calls	2	73
file_editor calls	507	1,004
terminal calls	1,978	5,274

Verdict

No hidden causes. The same 3 failure mechanisms account for 100% of stuck conversations, and all 3 also occur in successful conversations. The difference is entirely about recovery rate — successful conversations recover from these errors and continue, while stuck ones hit 3 consecutive failures and get killed by the stuck detector.

[VascoSch92]

@juanmichelini

I compared your issue with my issue.

Now I have a complete picture. Let me write the final summary:

Yes, These Issues Are Related! Both issues share the exact same root cause mechanism:

The Common Pattern

Issue	Model	Problem
#2482 (Claude Sonnet/Opus 4.5/4.6)	Anthropic extended thinking	Model puts reasoning in thinking_blocks but no text/tool_calls → empty MessageEvent
#714 (Qwen3.5-Flash)	Qwen thinking/reasoning	Model puts tool calls inside reasoning_content as XML (<tool_call>) but no actual function_call → empty MessageEvent

The Stuck Mechanism (Identical)

Empty MessageEvent #1 → Empty MessageEvent #2 → Empty MessageEvent #3 → STUCK!

Both models produce responses where:

• ✅ There IS reasoning/thinking content
• ❌ There is NO actionable output (no text, no tool_calls)
• ❌ The SDK treats this as "empty" and emits a useless MessageEvent
• ❌ After 3 consecutive empty messages → stuck

Key Difference

Aspect	Anthropic (Issue #2482)	Qwen (Issue #714)
Where reasoning goes	thinking_blocks	reasoning_content
Tool call format	N/A (no tools in thinking)	XML: <tool_call><function=...>
Is tool call recoverable?	No - model just didn't call tools	YES - tool call IS there, just in wrong field
Recovery rate	Unknown	76% in successful, 16% in stuck

The Qwen Issue is Actually Worse

For Qwen3.5-Flash, the tool call IS present in the response - it's just in reasoning_content instead of as a proper function_call. This is a parsing/extraction issue that litellm could potentially fix:

{
  "content": [{"text": ""}],  // empty!
  "reasoning_content": "Let me look at skipping.py:\n\n<tool_call>\n<function=file_editor>\n<parameter=command>view</parameter>\n<parameter=path>/workspace/pytest/src/_pytest</parameter>\n</function>\n</tool_call>"
}

Potential litellm Fix for Qwen

litellm could add a fallback parser that:

1. Detects <tool_call> XML in reasoning_content
2. Extracts and converts it to a proper tool_calls array
3. Returns a valid response with tool calls

This would be similar to how some providers handle tool calls in text content.

Summary Table

Root Cause	Anthropic Issue	Qwen Issue
Model produces reasoning-only response	✅	✅
No actionable content (text/tool_calls)	✅	✅
SDK emits empty MessageEvent	✅	✅
Stuck after 3 consecutive	✅	✅
Tool call is recoverable from reasoning	❌	✅
Could be fixed in litellm	❌	✅

Conclusion: Same mechanism, same symptom, but the Qwen issue has a potential litellm-side fix (parse tool calls from reasoning_content), while the Anthropic issue is more fundamental (model behavior with no workaround except SDK changes).