test: verify response_format and tools coexist in requests (#411)#429
test: verify response_format and tools coexist in requests (#411)#429robert-j-y wants to merge 6 commits intomainfrom
Conversation
Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
…enai behavior) Reverts the unconditional stripping of response_format when tools are present. The reference @ai-sdk/openai provider sends both fields together, and stripping breaks Output.object() + tools multi-step workflows where the model needs the JSON schema instruction in the final text-response step. Model-specific conflicts should be handled at the API adapter layer. Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
Addressing "Items for human review"1. Test-only PR: zero changes to production codeConfirmed correct. The net diff has zero changes to 2. Issue #411 remains partially unaddressedAcknowledged. The root cause (some models confuse 3. E2e test uses live API (
|
Description
Addresses #411. After investigation, the original approach of stripping
response_formatwhen tools are present was reverted because it breaks legitimateOutput.object()+ tools multi-step workflows.Key finding: The reference
@ai-sdk/openaiprovider sends bothresponse_formatandtoolstogether — it does not strip either field. Unconditionally strippingresponse_formatwould causeOutput.object()parsing to fail in the final text-response step of multi-step tool workflows, since the model would have no JSON schema instruction.What this PR does:
response_formatandtoolsare sent together, matching@ai-sdk/openaibehaviortool_callswork alongsideOutput.object()toHavePropertyassertionsWhat this PR does NOT do:
src/chat/index.ts— the existing behavior of sending both fields is already correctgenerateTextwithOutput.object()+tools- model returns tool call arguments as text instead of structuredtool_calls#411 where some models dump tool args as text — that should be handled at the API adapter layer per-modelUpdates since last revision
pnpm changeset --empty) since there are no source code changes — no version bump will be triggeredgenerateTextwithOutput.object()+tools- model returns tool call arguments as text instead of structuredtool_calls#411 remains partially unaddressed: The root cause (some models confusingresponse_format+tools) should be fixed at the OpenRouter API adapter layer for affected models, not in the SDK.openai/gpt-4o-minivia OpenRouter — verify this is acceptable for CI stability.Checklist
pnpm stylecheckandpnpm typecheckpnpm testand all tests pass (244/244)Changeset
pnpm changeset --empty(test-only, no release needed)Link to Devin session: https://app.devin.ai/sessions/05ec3186bbc44181810cf240b9bccdc4
Requested by: @robert-j-y