Skip to content

feat: merge undersized heading sections during chunking (chunking.min_chunk_size) - #609

Draft
j-sperling wants to merge 3 commits into
zilliztech:mainfrom
j-sperling:feat/chunker-min-chunk-merge
Draft

feat: merge undersized heading sections during chunking (chunking.min_chunk_size)#609
j-sperling wants to merge 3 commits into
zilliztech:mainfrom
j-sperling:feat/chunker-min-chunk-merge

Conversation

@j-sperling

@j-sperling j-sperling commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Heading-dense markdown (conversation transcripts, session logs — memsearch's primary memory shape) currently produces one chunk per heading section no matter how tiny; a 19,195-session corpus chunked into 186,192 pieces (~10x expansion), and embedding cost scales linearly with chunk count. This adds chunking.min_chunk_size: consecutive sections below the threshold are accumulated into one chunk (never exceeding max_chunk_size), carrying the first section's heading and the full merged line span.
  • Default 0 = exactly today's behavior (every section flushes immediately); the merge path only activates when opted in. Oversized sections still route through the existing paragraph splitter unchanged.
  • Plumbing: ChunkingConfig.min_chunk_size (TOML + config set), allowed in project-local .memsearch.toml (same trust tier as the existing chunking.max_chunk_size/overlap_lines), passed through MemSearch.__init__ and the CLI override map.
  • Trade-off note for review: merging changes chunk identity (chunk IDs hash content + line span), so enabling it on an existing collection re-embeds affected files once. Retrieval-quality impact is corpus-dependent; on a LongMemEval-S bench harness, coarser session-level granularity scored slightly better than per-turn chunks with the same embedder, but a maintainer-side eval would be worth running before recommending a non-zero default.

Test plan

  • New tests: default keeps per-section behavior; tiny sections merge (heading/line-span/order assertions); merge respects max_chunk_size; pending chunks flush before an oversized section; accumulation stops once the threshold is reached
  • Updated _cfg_to_memsearch_kwargs translation test and manual-construction test stubs for the new attribute
  • Full suite: 241 passed, 7 skipped
  • ruff check / ruff format --check clean

Design notes

  • Merging is heading-level-blind by design. Accumulation crosses heading levels (a small ### tail can merge into the following ## section). For the transcript shape that motivates this feature, levels carry no hierarchy (every turn is the same level), and stopping at level boundaries would reintroduce the tiny-chunk problem exactly where it bites. If level-aware merging turns out to matter for prose docs, it can be layered on later without changing the config surface.
  • Merged chunks keep the first section's heading in metadata only. All subsequent headings survive verbatim inside content (sections include their heading lines), so no text is lost to search; the metadata heading is used for display/filtering and the first section's heading is the least surprising representative for a run of consecutive tiny sections.

Heading-dense documents such as conversation transcripts produce one
tiny chunk per heading (a 19K-session corpus chunked to 186K pieces,
~10x), multiplying embedding cost linearly. Add chunking.min_chunk_size:
consecutive sections below the threshold accumulate into a single chunk
(never past max_chunk_size) carrying the first section's heading.

Default 0 preserves the existing one-chunk-per-section behavior exactly.
The knob is config-file settable, project-local allowed (same trust tier
as max_chunk_size), and plumbed through MemSearch and the CLI overrides.
…unk-merge

# Conflicts:
#	src/memsearch/cli.py
#	src/memsearch/config.py
#	src/memsearch/core.py
#	tests/test_cli_config_helpers.py
#	tests/test_embed_batching.py
The chunking.min_chunk_size _PARAM_MAP entry had no corresponding Click
option, leaving the knob TOML-only. Add the flag to index and watch
mirroring --max-chunk-size, and document the key in the CLI/config/API
references, including the caveat that enabling merging changes chunk
IDs and triggers a one-time re-embed of affected files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant