Skip to content

test: verify response_format and tools coexist in requests (#411)#429

Open
robert-j-y wants to merge 6 commits intomainfrom
devin/1772828576-fix-output-object-tools-conflict
Open

test: verify response_format and tools coexist in requests (#411)#429
robert-j-y wants to merge 6 commits intomainfrom
devin/1772828576-fix-output-object-tools-conflict

Conversation

@robert-j-y
Copy link
Copy Markdown
Contributor

@robert-j-y robert-j-y commented Mar 6, 2026

Description

Addresses #411. After investigation, the original approach of stripping response_format when tools are present was reverted because it breaks legitimate Output.object() + tools multi-step workflows.

Key finding: The reference @ai-sdk/openai provider sends both response_format and tools together — it does not strip either field. Unconditionally stripping response_format would cause Output.object() parsing to fail in the final text-response step of multi-step tool workflows, since the model would have no JSON schema instruction.

What this PR does:

  • Adds explicit unit tests (doGenerate + doStream) verifying both response_format and tools are sent together, matching @ai-sdk/openai behavior
  • Adds an e2e regression test confirming structured tool_calls work alongside Output.object()
  • Strengthens the existing streaming test with explicit toHaveProperty assertions

What this PR does NOT do:

Updates since last revision

  • Changed changeset to empty (pnpm changeset --empty) since there are no source code changes — no version bump will be triggered

⚠️ Items for human review

Checklist

  • I have run pnpm stylecheck and pnpm typecheck
  • I have run pnpm test and all tests pass (244/244)
  • I have added tests for my changes (if applicable)
  • I have updated documentation (if applicable)

Changeset

  • I have run pnpm changeset --empty (test-only, no release needed)

Link to Devin session: https://app.devin.ai/sessions/05ec3186bbc44181810cf240b9bccdc4
Requested by: @robert-j-y


Open with Devin

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.

Open in Devin Review

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
devin-ai-integration[bot]

This comment was marked as resolved.

@robert-j-y robert-j-y requested a review from mattapperson March 7, 2026 00:07
…enai behavior)

Reverts the unconditional stripping of response_format when tools are present.
The reference @ai-sdk/openai provider sends both fields together, and stripping
breaks Output.object() + tools multi-step workflows where the model needs the
JSON schema instruction in the final text-response step.

Model-specific conflicts should be handled at the API adapter layer.

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
@robert-j-y robert-j-y changed the title fix: omit response_format when tools are present (#411) fix: ensure response_format and tools coexist in requests (#411) Mar 7, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
@robert-j-y robert-j-y changed the title fix: ensure response_format and tools coexist in requests (#411) test: verify response_format and tools coexist in requests (#411) Mar 7, 2026
@robert-j-y
Copy link
Copy Markdown
Contributor Author

Addressing "Items for human review"

1. Test-only PR: zero changes to production code

Confirmed correct. The net diff has zero changes to src/chat/index.ts — the stripping logic was added in commit 3663173 then reverted in cdae6a5. The empty changeset (pnpm changeset --empty) will not trigger a version bump or CHANGELOG entry. This PR adds valuable regression test coverage ensuring both response_format and tools coexist in requests, matching @ai-sdk/openai behavior.

2. Issue #411 remains partially unaddressed

Acknowledged. The root cause (some models confuse response_format + tools and dump tool args as text) is model-specific behavior that should be handled at the OpenRouter API adapter layer per-model, not universally in the SDK. The SDK correctly sends both fields, matching the reference provider. This is documented in the PR description.

3. E2e test uses live API (openai/gpt-4o-mini)

Acceptable. openai/gpt-4o-mini is one of the most stable and widely available models on OpenRouter. The test verifies structured tool_calls work alongside Output.object(), which is a fundamental workflow. CI has passed consistently with this model.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant