Skip to content

[Bug] 知识库分块:RecursiveCharacterChunker 的 overlap 重复计长导致输出块超限,embedding 数量失真 #9901

Description

@xiaoyuyu6420

问题描述

RecursiveCharacterChunker 在块重叠(overlap)处理上会产出超过 chunk_size 上限的块。

astrbot/core/knowledge_base/chunking/recursive.py(约 108-123 行):当一个 split 会导致当前块超过 chunk_size 时,代码先 flush 当前块,然后取 combined_text[-overlap:] 作为 overlap_text,并直接构造 current_chunk = [overlap_text, split]current_chunk_length = len(overlap_text) + split_length——没有重新检查这个新累计长度是否已经超过 chunk_size。因此下一个 flush 出来的块实际是 chunk_size + overlap 的长度(默认 512/50 下每块约 562 字符,超过配置上限)。

而同一模块的 _split_by_character(约 160-161 行)反而有 overlap >= chunk_size 时抛异常的防护,两条路径行为不一致。

影响

  • 默认 RecursiveCharacterChunker(默认分块器)入库的所有文档,分块均超过配置上限
  • 用户在 dashboard 调大 overlap 时,每块中 overlap 部分占比畸高,重复内容被反复嵌入,embedding 数量与检索质量都受影响

建议修复

flush 并追加 overlap_text 后,重新计算累计长度并检查是否超过 chunk_size;若超过则按独立块处理,或与 _split_by_character 保持一致的防护逻辑。

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions