Skip to content

test: add regression test for issue #248 (Gemini web search empty responses)#377

Merged
robert-j-y merged 4 commits intomainfrom
devin/1769563153-issue-248-gemini-web-search
Jan 28, 2026
Merged

test: add regression test for issue #248 (Gemini web search empty responses)#377
robert-j-y merged 4 commits intomainfrom
devin/1769563153-issue-248-gemini-web-search

Conversation

@robert-j-y
Copy link
Contributor

@robert-j-y robert-j-y commented Jan 28, 2026

Description

Adds regression tests for issue #248 (Gemini 3 Pro Preview returning empty responses when using web search plugin).

Test coverage for all models mentioned in the issue thread:

  • google/gemini-3-pro-preview (original issue): generateText, streamText, and part number search queries
  • google/gemini-3-flash-preview (Jan 27 comment: "works")
  • anthropic/claude-sonnet-4 (Dec 7 comment: stream cuts, partial response)
  • anthropic/claude-opus-4 (Dec 7 comment: bugs out, no error)
  • openai/gpt-4.1 (Dec 7 comment: "only consistent experience")

Configuration: Tests use the exact code pattern from the issue report with providerOptions: { openrouter: { reasoning: { effort: 'high' }, plugins: [{ id: 'web' }] } }.

Updates since last revision

  • Expanded test coverage to include all models mentioned in the issue thread (7 tests total across 5 models)
  • Removed root cause speculation from comments; test header now contains only factual timeline from the issue thread
  • Note: The part number search test may fail, documenting the actual current bug state

Human Review Checklist

  • Verify the test code pattern matches the original issue report exactly (uses providerOptions not model-level settings)
  • The part number test ("602-3973-01") covers the specific scenario from the Dec 7 comment
  • Model names in tests match what was mentioned in the issue thread
  • Test comments contain only facts from the issue, no speculation

Checklist

  • I have run pnpm stylecheck and pnpm typecheck
  • I have run pnpm test and all tests pass
  • I have added tests for my changes (if applicable)
  • I have updated documentation (if applicable) - N/A

Changeset

  • I have run pnpm changeset to create a changeset file

Note: Empty changeset used since this is a test-only change that doesn't need a release.


Link to Devin run: https://app.devin.ai/sessions/b246f84ecad04155bed9b1af8a649f56
Requested by: Robert Yeakel (@robert-j-y)

devin-ai-integration bot and others added 4 commits January 28, 2026 01:22
…ponses)

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
… thread

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
@robert-j-y robert-j-y merged commit 66b68af into main Jan 28, 2026
2 checks passed
@robert-j-y robert-j-y deleted the devin/1769563153-issue-248-gemini-web-search branch January 28, 2026 02:14
kesavan-byte pushed a commit to osm-API/ai-sdk-provider that referenced this pull request Feb 13, 2026
…arch empty responses) (OpenRouterTeam#377)

* test: add regression test for issue OpenRouterTeam#248 (Gemini web search empty responses)

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

* fix: add reasoning config to match exact issue OpenRouterTeam#248 scenario

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

* fix: update test to use exact providerOptions pattern from issue OpenRouterTeam#248

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

* test: add comprehensive tests for issue OpenRouterTeam#248 covering all models from thread

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments