Long-context chunking can recurse indefinitely on CJK-heavy input after 400 maximum context length

[wllbll]

Plugin Version

1.1.0-beta.9

OpenClaw Version

3.13

Bug Description

Summary

When embedding very long CJK-heavy text, memory-lancedb-pro can enter an infinite chunking/retry loop after the provider returns:

400 Invalid 'input': maximum context length is 8192 tokens.

This can quickly cause a request storm.

Environment

• Repo: CortexReach/memory-lancedb-pro
• Version tested: 1.1.0-beta.9
• Branch/commit tested: master / 2ebba8e6b7b65bf38336199384d5ec8690701f6e
• Embedding model: text-embedding-3-small
• Provider: OpenAI-compatible API
• Config:

  {
    "embedding": {
      "model": "text-embedding-3-small",
      "chunking": true
    }
  }
  

What happens

For some CJK-heavy inputs, the first embedding request correctly fails with a context-length error, and the plugin starts chunking.

However, one of the generated chunks can still be too large for the provider in token terms, while smartChunk() returns exactly 1 chunk of the same size for that chunk.

That leads to repeated recursion:

• embed oversized chunk
• get 400 maximum context length
• chunk again
• still get 1 chunk with the same size
• recurse again

Expected behavior

If chunking does not actually reduce the failing chunk, the plugin should stop and return an error instead of recursively retrying.

Suspected root cause

src/embedder.ts retries chunking on context errors, but there is no guard for:

• chunkResult.chunks.length === 1
• and the chunk is effectively unchanged from the original failing input

This seems especially easy to trigger with CJK-heavy text because character-count heuristics are not a safe proxy for token count.

Suggested fix

Before recursively retrying chunk embeddings, add a guard like:

• if chunk count is 1
• and the chunk length is unchanged (or not meaningfully reduced)

then abort and throw the original context-length error instead of retrying again.

Additional note

This is not just a "quota exceeded" symptom. In our testing, the initial trigger is a real provider 400 maximum context length, and the request storm is the secondary
effect.

Expected Behavior

If chunking does not actually reduce the failing chunk, the plugin should stop and return an error instead of recursively retrying.

Steps to Reproduce

Reproduction notes

Input that reproduced reliably:

• "啊".repeat(12000)

Observed chunking behavior from smartChunk(text, "text-embedding-3-small"):

{"len":12000,"chunkCount":3,"chunkLengths":[5734,5734,1350]}
{"len":5734,"chunkCount":1,"chunkLengths":[5734]}

This appears to be the core issue:

• 5734 chars may still exceed the provider's token limit for CJK-heavy text
• but chunking 5734 chars returns the same single 5734-char chunk
• so the embedder retries recursively forever

Error Logs / Screenshots

## Minimal observed behavior

  Real provider logs:

  Document exceeded context limit (400 Invalid 'input': maximum context length is 8192 tokens.), attempting chunking...
  Split document into 3 chunks for embedding
  Document exceeded context limit (400 Invalid 'input': maximum context length is 8192 tokens.), attempting chunking...
  Split document into 1 chunks for embedding
  Document exceeded context limit (400 Invalid 'input': maximum context length is 8192 tokens.), attempting chunking...
  Split document into 1 chunks for embedding

  In a 5-second controlled test, this produced:

  - 75 embedding requests
  - repeated request lengths only among:
      - 12000
      - 5734
      - 1350

Embedding Provider

OpenAI

OS / Platform

Ubuntu

[Hi-Jiajun]

遇到了同样的情况

[ZerlenZhang]

我也遇到了，感觉很容易触发——然后我让我的龙虾去修，似乎这样就能修好，目前为止没有遇到了

好的！这是修改的 diff：

--- a/src/embedder.ts
+++ b/src/embedder.ts
@@ -466,7 +466,7 @@ export class Embedder {
   // --------------------------------------------------------------------------

-  private async embedSingle(text: string, task?: string): Promise<number[]> {
+  private async embedSingle(text: string, task?: string, _allowChunking: boolean = true): Promise<number[]> {
     if (!text || text.trim().length === 0) {
       throw new Error("Cannot embed empty text");
     }
@@ -494,6 +494,10 @@ export class Embedder {
           if (chunkResult.chunks.length === 0) {
             throw new Error(`Failed to chunk document: ${errorMsg}`);
           }
+
+          // If only 1 chunk and still in chunking mode, don't recursively chunk again
+          // to prevent infinite loop when the chunk itself exceeds context limit
+          const canRecursivelyChunk = chunkResult.chunks.length > 1;

           // Embed all chunks in parallel
           console.log(`Split document into ${chunkResult.chunkCount} chunks for embedding`);
           const chunkEmbeddings = await Promise.all(
@@ -501,7 +505,7 @@ export class Embedder {
               try {
-                const embedding = await this.embedSingle(chunk, task);
+                // Pass false to prevent recursive chunking on individual chunks
+                const embedding = await this.embedSingle(chunk, task, canRecursivelyChunk);
                 return { embedding };
               } catch (chunkError) {
                 console.warn(`Failed to embed chunk ${idx}:`, chunkError);
@@ -612,6 +616,9 @@ export class Embedder {
                 throw new Error("Chunker produced no chunks");
               }

+              // If only 1 chunk, don't recursively chunk to prevent infinite loop
+              const canRecursivelyChunk = chunkResult.chunks.length > 1;
+
               // Embed all chunks in parallel, then average.
               const embeddings = await Promise.all(
-                chunkResult.chunks.map((chunk) => this.embedSingle(chunk, task))
+                chunkResult.chunks.map((chunk) => this.embedSingle(chunk, task, canRecursivelyChunk))
               );

核心思路：

• 添加 _allowChunking 参数控制是否允许递归 chunking
• 当 chunking 后只剩 1 个 chunk 时，设为 false 防止再次触发 chunking
• 这样即使单个 chunk 仍然超长，也会直接抛出错误而不是无限循环

[AliceLJY]

感谢报告！这个 bug 跟 #214 是同一个问题——chunking 产出单个 chunk 时无限递归。

已有两个 PR 在修：

• PR fix: prevent infinite recursion in embedSingle() for CJK text #215 — 基础修复（防止单 chunk 递归）
• PR fix: prevent infinite recursion in embedSingle() for CJK text (replaces PR #215) #238 — follow-up，补充 timeout 和收敛性保障

@Hi-Jiajun @ZerlenZhang 你们遇到的情况应该都能被这两个 PR 覆盖。修复合并后会关闭此 issue。

[AliceLJY]

Closing as duplicate of #214. Fix tracked in PR #215 and #238.