feat: add Anthropic Batch API support for 50% cost savings #8668
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add anthropicUseBatchApi boolean toggle to provider settings schema - Implement batch API methods (create job, poll status, retrieve results) - Add clean progress indicators during batch processing (every 20s) - Add 10-minute timeout handling for batch jobs - Process requests asynchronously with 50% cost discount - Add UI toggle in Anthropic provider settings - Add comprehensive tests for batch API functionality - All existing features (prompt caching, 1M context) remain compatible Resolves #8667
dosubot
bot
added
size:XL
2 tasks
![roomote[bot]](https://avatars.githubusercontent.com/in/1546624?s=60&v=4){kind=small}
Comment on lines
+75
to
+94
| private async createBatchJob(requests: BatchRequest[]): Promise<string> { | ||
| const response = await fetch(`${this.client.baseURL || "https://api.anthropic.com"}/v1/messages/batches`, { | ||
| method: "POST", | ||
| headers: { | ||
| "Content-Type": "application/json", | ||
| "x-api-key": this.options.apiKey || "", | ||
| "anthropic-version": "2023-06-01", | ||
| "anthropic-beta": "message-batches-2024-09-24", | ||
| }, | ||
| body: JSON.stringify({ requests }), | ||
| }) | ||
|
|
||
| if (!response.ok) { | ||
| const error = await response.text() | ||
| throw new Error(`Failed to create batch job: ${error}`) | ||
| } | ||
|
|
||
| const job: BatchJob = await response.json() | ||
| return job.id | ||
| } |
Author
There was a problem hiding this comment.
The batch API methods don't respect the anthropicUseAuthToken setting. When users configure a custom base URL with auth token mode (by setting both anthropicBaseUrl and anthropicUseAuthToken), the SDK constructor uses authToken instead of apiKey (line 68), but the batch API methods always hardcode x-api-key header.
This inconsistency will cause authentication failures for users with custom Anthropic-compatible endpoints that require Authorization headers. The batch API methods should use the same authentication logic as the SDK client.
Suggested change
| private async createBatchJob(requests: BatchRequest[]): Promise<string> { | |
| const response = await fetch(`${this.client.baseURL || "https://api.anthropic.com"}/v1/messages/batches`, { | |
| method: "POST", | |
| headers: { | |
| "Content-Type": "application/json", | |
| "x-api-key": this.options.apiKey || "", | |
| "anthropic-version": "2023-06-01", | |
| "anthropic-beta": "message-batches-2024-09-24", | |
| }, | |
| body: JSON.stringify({ requests }), | |
| }) | |
| if (!response.ok) { | |
| const error = await response.text() | |
| throw new Error(`Failed to create batch job: ${error}`) | |
| } | |
| const job: BatchJob = await response.json() | |
| return job.id | |
| } | |
| private async createBatchJob(requests: BatchRequest[]): Promise<string> { | |
| const apiKeyFieldName = | |
| this.options.anthropicBaseUrl && this.options.anthropicUseAuthToken ? "Authorization" : "x-api-key" | |
| const apiKeyValue = | |
| this.options.anthropicBaseUrl && this.options.anthropicUseAuthToken | |
| ? `Bearer ${this.options.apiKey || ""}` | |
| : this.options.apiKey || "" | |
| const response = await fetch(`${this.client.baseURL || "https://api.anthropic.com"}/v1/messages/batches`, { | |
| method: "POST", | |
| headers: { | |
| "Content-Type": "application/json", | |
| [apiKeyFieldName]: apiKeyValue, | |
| "anthropic-version": "2023-06-01", | |
| "anthropic-beta": "message-batches-2024-09-24", | |
| }, | |
| body: JSON.stringify({ requests }), | |
| }) | |
| if (!response.ok) { | |
| const error = await response.text() | |
| throw new Error(`Failed to create batch job: ${error}`) | |
| } | |
| const job: BatchJob = await response.json() | |
| return job.id | |
| } |
Comment on lines
+99
to
+118
| private async getBatchJobStatus(jobId: string): Promise<BatchJob> { | ||
| const response = await fetch( | ||
| `${this.client.baseURL || "https://api.anthropic.com"}/v1/messages/batches/${jobId}`, | ||
| { | ||
| method: "GET", | ||
| headers: { | ||
| "x-api-key": this.options.apiKey || "", | ||
| "anthropic-version": "2023-06-01", | ||
| "anthropic-beta": "message-batches-2024-09-24", | ||
| }, | ||
| }, | ||
| ) | ||
|
|
||
| if (!response.ok) { | ||
| const error = await response.text() | ||
| throw new Error(`Failed to get batch job status: ${error}`) | ||
| } | ||
|
|
||
| return response.json() | ||
| } |
Author
There was a problem hiding this comment.
Same authentication issue as createBatchJob - this method also needs to respect the anthropicUseAuthToken setting for consistency with custom base URL configurations.
Suggested change
| private async getBatchJobStatus(jobId: string): Promise<BatchJob> { | |
| const response = await fetch( | |
| `${this.client.baseURL || "https://api.anthropic.com"}/v1/messages/batches/${jobId}`, | |
| { | |
| method: "GET", | |
| headers: { | |
| "x-api-key": this.options.apiKey || "", | |
| "anthropic-version": "2023-06-01", | |
| "anthropic-beta": "message-batches-2024-09-24", | |
| }, | |
| }, | |
| ) | |
| if (!response.ok) { | |
| const error = await response.text() | |
| throw new Error(`Failed to get batch job status: ${error}`) | |
| } | |
| return response.json() | |
| } | |
| private async getBatchJobStatus(jobId: string): Promise<BatchJob> { | |
| const apiKeyFieldName = | |
| this.options.anthropicBaseUrl && this.options.anthropicUseAuthToken ? "Authorization" : "x-api-key" | |
| const apiKeyValue = | |
| this.options.anthropicBaseUrl && this.options.anthropicUseAuthToken | |
| ? `Bearer ${this.options.apiKey || ""}` | |
| : this.options.apiKey || "" | |
| const response = await fetch( | |
| `${this.client.baseURL || "https://api.anthropic.com"}/v1/messages/batches/${jobId}`, | |
| { | |
| method: "GET", | |
| headers: { | |
| [apiKeyFieldName]: apiKeyValue, | |
| "anthropic-version": "2023-06-01", | |
| "anthropic-beta": "message-batches-2024-09-24", | |
| }, | |
| }, | |
| ) | |
| if (!response.ok) { | |
| const error = await response.text() | |
| throw new Error(`Failed to get batch job status: ${error}`) | |
| } | |
| return response.json() | |
| } |
Comment on lines
+123
to
+143
| private async getBatchResults(jobId: string): Promise<BatchResult[]> { | ||
| const response = await fetch( | ||
| `${this.client.baseURL || "https://api.anthropic.com"}/v1/messages/batches/${jobId}/results`, | ||
| { | ||
| method: "GET", | ||
| headers: { | ||
| "x-api-key": this.options.apiKey || "", | ||
| "anthropic-version": "2023-06-01", | ||
| "anthropic-beta": "message-batches-2024-09-24", | ||
| }, | ||
| }, | ||
| ) | ||
|
|
||
| if (!response.ok) { | ||
| const error = await response.text() | ||
| throw new Error(`Failed to get batch results: ${error}`) | ||
| } | ||
|
|
||
| const data = await response.json() | ||
| return data.results || [] | ||
| } |
Author
There was a problem hiding this comment.
Same authentication issue as the other batch API methods - needs to respect the anthropicUseAuthToken setting.
Suggested change
| private async getBatchResults(jobId: string): Promise<BatchResult[]> { | |
| const response = await fetch( | |
| `${this.client.baseURL || "https://api.anthropic.com"}/v1/messages/batches/${jobId}/results`, | |
| { | |
| method: "GET", | |
| headers: { | |
| "x-api-key": this.options.apiKey || "", | |
| "anthropic-version": "2023-06-01", | |
| "anthropic-beta": "message-batches-2024-09-24", | |
| }, | |
| }, | |
| ) | |
| if (!response.ok) { | |
| const error = await response.text() | |
| throw new Error(`Failed to get batch results: ${error}`) | |
| } | |
| const data = await response.json() | |
| return data.results || [] | |
| } | |
| private async getBatchResults(jobId: string): Promise<BatchResult[]> { | |
| const apiKeyFieldName = | |
| this.options.anthropicBaseUrl && this.options.anthropicUseAuthToken ? "Authorization" : "x-api-key" | |
| const apiKeyValue = | |
| this.options.anthropicBaseUrl && this.options.anthropicUseAuthToken | |
| ? `Bearer ${this.options.apiKey || ""}` | |
| : this.options.apiKey || "" | |
| const response = await fetch( | |
| `${this.client.baseURL || "https://api.anthropic.com"}/v1/messages/batches/${jobId}/results`, | |
| { | |
| method: "GET", | |
| headers: { | |
| [apiKeyFieldName]: apiKeyValue, | |
| "anthropic-version": "2023-06-01", | |
| "anthropic-beta": "message-batches-2024-09-24", | |
| }, | |
| }, | |
| ) | |
| if (!response.ok) { | |
| const error = await response.text() | |
| throw new Error(`Failed to get batch results: ${error}`) | |
| } | |
| const data = await response.json() | |
| return data.results || [] | |
| } |
hannesrudolph
added
the
Issue/PR - Triage
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
| "maxTokensDescription": "Claude Code yanıtları için maksimum çıktı token sayısı. Varsayılan 8000'dir." | ||
| } | ||
| }, | ||
| "anthropicBatchApiLabel": "Use Batch API (50% cost savings)", |
There was a problem hiding this comment.
New keys 'anthropicBatchApiLabel' and 'anthropicBatchApiDescription' are added with English text. Consider providing Turkish translations to ensure locale consistency.
Suggested change
| "anthropicBatchApiLabel": "Use Batch API (50% cost savings)", | |
| "anthropicBatchApiLabel": "Toplu API'yi Kullan (%50 maliyet tasarrufu)", |
This comment was generated because it violated a code review rule: irule_C0ez7Rji6ANcGkkX.
| "maxTokensDescription": "Số lượng token đầu ra tối đa cho các phản hồi của Claude Code. Mặc định là 8000." | ||
| } | ||
| }, | ||
| "anthropicBatchApiLabel": "Use Batch API (50% cost savings)", |
There was a problem hiding this comment.
The added keys 'anthropicBatchApiLabel' and 'anthropicBatchApiDescription' remain in English. Please consider localizing them into Vietnamese for consistency.
Suggested change
| "anthropicBatchApiLabel": "Use Batch API (50% cost savings)", | |
| "anthropicBatchApiLabel": "Sử dụng Batch API (tiết kiệm 50% chi phí)", |
This comment was generated because it violated a code review rule: irule_C0ez7Rji6ANcGkkX.
| "maxTokensDescription": "Claude Code 响应的最大输出 Token 数量。默认为 8000。" | ||
| } | ||
| }, | ||
| "anthropicBatchApiLabel": "Use Batch API (50% cost savings)", |
There was a problem hiding this comment.
The new keys 'anthropicBatchApiLabel' and 'anthropicBatchApiDescription' are still in English. Consider providing proper Simplified Chinese translations.
Suggested change
| "anthropicBatchApiLabel": "Use Batch API (50% cost savings)", | |
| "anthropicBatchApiLabel": "使用批量 API（节省 50% 成本）", |
This comment was generated because it violated a code review rule: irule_C0ez7Rji6ANcGkkX.
| "maxTokensDescription": "Claude Code 回應的最大輸出 Token 數量。預設為 8000。" | ||
| } | ||
| }, | ||
| "anthropicBatchApiLabel": "Use Batch API (50% cost savings)", |
There was a problem hiding this comment.
The new translation keys 'anthropicBatchApiLabel' and 'anthropicBatchApiDescription' are provided in English. Please update them to Traditional Chinese.
Suggested change
| "anthropicBatchApiLabel": "Use Batch API (50% cost savings)", | |
| "anthropicBatchApiLabel": "使用批次 API（節省 50% 成本）", |
This comment was generated because it violated a code review rule: irule_C0ez7Rji6ANcGkkX.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements support for Anthropic's Batch API, which processes requests asynchronously with 50% cost savings compared to standard API calls. This addresses Issue #8667.
Key Features
✅ 50% Cost Reduction: Process requests at half the price when time is not critical
✅ Simple Toggle: Enable/disable Batch API with a single checkbox in settings
✅ Clean Progress Indicators: Non-spammy updates every 20 seconds during processing
✅ Graceful Timeout Handling: 10-minute maximum wait time with proper error handling
✅ Full Compatibility: Works with all existing features (prompt caching, 1M context)
✅ Zero Breaking Changes: Default behavior unchanged, fully backward compatible
Implementation Details
Settings
anthropicUseBatchApiboolean toggle to provider settings schemaCore Functionality
Testing
Code Quality
User Benefit
Users working on non-urgent tasks (documentation, refactoring, analysis) can cut their Anthropic API costs in half by simply toggling a setting. This makes Roo Code significantly more cost-effective for budget-conscious developers.
How It Works
When Batch API is enabled:
Testing Instructions
Closes #8667
Important
Adds support for Anthropic's Batch API with a 50% cost reduction, including settings toggle, progress indicators, and comprehensive testing.
anthropic.ts, allowing asynchronous request processing with 50% cost savings.anthropicUseBatchApiin settings to enable/disable Batch API.anthropicUseBatchApiboolean toggle to provider settings schema inprovider-settings.ts.Anthropic.tsxto include a toggle for Batch API with description and translations.anthropic.spec.tsfor batch processing flow, timeout, failure handling, progress updates, and cost calculation.This description was created by
for b783f43. You can customize this summary. It will automatically update as commits are pushed.