Skip to content

feat: add embedding rate limiting to prevent API quota exhaustion For Issue #5713 #5715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
SatoshiReport wants to merge 1 commit into from
Closed

feat: add embedding rate limiting to prevent API quota exhaustion For Issue #5713 #5715

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/services/code-index/constants/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,6 @@ export const BATCH_PROCESSING_CONCURRENCY = 10

/**Gemini Embedder */
export const GEMINI_MAX_ITEM_TOKENS = 2048

/**Embedding Rate Limiting */
export const EMBEDDING_CALL_DELAY_MS = 100 // Pause between embedding API calls to prevent rate limiting
4 changes: 4 additions & 0 deletions src/services/code-index/processors/file-watcher.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import {
BATCH_SEGMENT_THRESHOLD,
MAX_BATCH_RETRIES,
INITIAL_RETRY_DELAY_MS,
EMBEDDING_CALL_DELAY_MS,
} from "../constants"
import { createHash } from "crypto"
import { RooIgnoreController } from "../../../core/ignore/RooIgnoreController"
Expand Down Expand Up @@ -542,6 +543,9 @@ export class FileWatcher implements IFileWatcher {
if (this.embedder && blocks.length > 0) {
const texts = blocks.map((block) => block.content)
const { embeddings } = await this.embedder.createEmbeddings(texts)

// Add pause between embedding calls to prevent rate limiting
await new Promise(resolve => setTimeout(resolve, EMBEDDING_CALL_DELAY_MS))

pointsToUpsert = blocks.map((block, index) => {
const normalizedAbsolutePath = generateNormalizedAbsolutePath(block.file_path, this.workspacePath)
Expand Down
4 changes: 4 additions & 0 deletions src/services/code-index/processors/scanner.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import {
INITIAL_RETRY_DELAY_MS,
PARSING_CONCURRENCY,
BATCH_PROCESSING_CONCURRENCY,
EMBEDDING_CALL_DELAY_MS,
} from "../constants"
import { isPathInIgnoredDirectory } from "../../glob/ignore-utils"
import { TelemetryService } from "@roo-code/telemetry"
Expand Down Expand Up @@ -344,6 +345,9 @@ export class DirectoryScanner implements IDirectoryScanner {

// Create embeddings for batch
const { embeddings } = await this.embedder.createEmbeddings(batchTexts)

// Add pause between embedding calls to prevent rate limiting
await new Promise(resolve => setTimeout(resolve, EMBEDDING_CALL_DELAY_MS))

// Prepare points for Qdrant
const points = batchBlocks.map((block, index) => {
Expand Down