Skip to content

test: add regression test for issue #234 prompt caching behavior#375

Merged
robert-j-y merged 6 commits intomainfrom
devin/1769563153-issue-234-caching-test
Jan 28, 2026
Merged

test: add regression test for issue #234 prompt caching behavior#375
robert-j-y merged 6 commits intomainfrom
devin/1769563153-issue-234-caching-test

Conversation

@robert-j-y
Copy link
Contributor

@robert-j-y robert-j-y commented Jan 28, 2026

Description

Adds a regression test for issue #234 ("Delayed caching with GPT 4.1, none with Kimi K2 Thinking") to the e2e/issues/ directory.

What the test verifies:

  • Provider correctly exposes cachedTokens in providerMetadata.openrouter.usage.promptTokensDetails when present
  • Provider metadata includes detailed usage information (promptTokens, completionTokens, totalTokens)
  • Response structure is valid regardless of whether caching occurs
  • For Kimi K2 (thinking model), also verifies reasoningTokens is correctly exposed

Test structure:

  • Makes 3 consecutive requests for both GPT 4.1 and Kimi K2 (matching the exact issue scenario)
  • Logs output in a concise single-line format per request
  • Test header documents only the known facts from the issue (no root cause speculation)

Updates since last revision

  • Removed redundant comments throughout the test file
  • Consolidated console.log output to single-line format per request
  • Test code is now minimal and focused on verification logic

Human Review Checklist

  • Verify test follows the e2e/issues/ pattern from PR test: add e2e/issues directory with issue #194 regression test #369
  • Confirm test header documents only known facts from the issue
  • Check if 180s timeout is reasonable for 6 API calls (3 GPT 4.1 + 3 Kimi K2)
  • Note: Tests verify structure, not caching behavior (caching is non-deterministic and depends on server routing)
  • Note: Tests depend on external APIs - may be flaky if models are unavailable

Checklist

  • I have run pnpm stylecheck and pnpm typecheck
  • I have run pnpm test and all tests pass
  • I have added tests for my changes (if applicable)
  • I have updated documentation (if applicable) - N/A

Changeset

  • I have run pnpm changeset to create a changeset file

Note: Empty changeset used since this is a test-only change that doesn't need a release.


Link to Devin run: https://app.devin.ai/sessions/7ed009fd989445428e2bd9f81b697781
Requested by: Robert Yeakel (@robert-j-y)

devin-ai-integration bot and others added 6 commits January 28, 2026 01:21
Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
… requests

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
…wn facts

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>
@robert-j-y robert-j-y merged commit 7df7404 into main Jan 28, 2026
2 checks passed
@robert-j-y robert-j-y deleted the devin/1769563153-issue-234-caching-test branch January 28, 2026 02:24
kesavan-byte pushed a commit to osm-API/ai-sdk-provider that referenced this pull request Feb 13, 2026
… behavior (OpenRouterTeam#375)

* test: add regression test for issue OpenRouterTeam#234 prompt caching behavior

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

* fix: use correct Kimi K2 model ID (moonshotai/kimi-k2-thinking)

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

* refactor: improve test robustness - match exact issue scenario with 3 requests

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

* refactor: match exact issue OpenRouterTeam#234 format with 3 requests for both models

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

* refactor: remove root cause analysis from test header - keep only known facts

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

* refactor: remove redundant comments from test file

Co-Authored-By: Robert Yeakel <robert.yeakel@openrouter.ai>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments