Updates Claude 3.7 model token limits

[vagadiya]

Context

Updates Claude 3.7 model token limits

Increases maxTokens from 8192 to 64000 for Claude 3.7 Sonnet model configurations across various API integrations including Anthropic, Bedrock, Glama, Requesty, and OpenRouter.

Also adjusts Vertex AI thinking model maxTokens to 128000 for improved performance.

Implementation

Configuration values have been increased to 64K for Claude 3.7

I noticed that they were 64K before, but were downgraded to 8K, any reason for that?

Get in Touch

Roo Code Discord : vagadiya

---

Important

Increase maxTokens for Claude 3.7 model to 64000 across various integrations and adjust Vertex AI thinking model to 128000.

• Behavior:
  • Increase maxTokens from 8192 to 64000 for Claude 3.7 Sonnet in anthropicModels, bedrockModels, glamaDefaultModelInfo, requestyDefaultModelInfo, and openRouterDefaultModelInfo in api.ts.
  • Adjust maxTokens to 128000 for Vertex AI thinking model in vertexModels in api.ts.
• Tests:
  • Update expected maxTokens to 64000 in anthropic.test.ts, glama.test.ts, requesty.test.ts, and vertex.test.ts.
  • Ensure tests reflect new token limits and verify correct behavior for Claude 3.7 models.

^{This description was created by for c3e22d5. It will automatically update as commits are pushed.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: c3e22d5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[mrubens]

Right now only the thinking variants are set up to handle the longer context windows / have a slider to change the values. We'd need to think more deeply about this change, since more max tokens isn't always better for people with Sonnet (I think in most cases 8-16k is the sweet spot).

[vagadiya]

Got it, understood. Closing this PR.

I thought the change would be useful to those directly using the model providers.

As per the other support ticket (https://discord.com/channels/1332146336664915968/1361732754433114203), I have been using a custom Roo build for Bedrock & custom-arn, with 64K max tokens. It has been doing well on our code bases to ensure large responses back don't get truncated, else was seeing tool errors and seems to be that the 8K was too little.

Regardless, agree that we probably need a slider based option for the mainstream model providers.

I'll work on the implementation for the custom-arn, to allow flexible configuration. Will take me a while though.