feat: merge undersized heading sections during chunking (chunking.min_chunk_size) - #609
Draft
j-sperling wants to merge 3 commits into
Draft
feat: merge undersized heading sections during chunking (chunking.min_chunk_size)#609j-sperling wants to merge 3 commits into
j-sperling wants to merge 3 commits into
Conversation
Heading-dense documents such as conversation transcripts produce one tiny chunk per heading (a 19K-session corpus chunked to 186K pieces, ~10x), multiplying embedding cost linearly. Add chunking.min_chunk_size: consecutive sections below the threshold accumulate into a single chunk (never past max_chunk_size) carrying the first section's heading. Default 0 preserves the existing one-chunk-per-section behavior exactly. The knob is config-file settable, project-local allowed (same trust tier as max_chunk_size), and plumbed through MemSearch and the CLI overrides.
…unk-merge # Conflicts: # src/memsearch/cli.py # src/memsearch/config.py # src/memsearch/core.py # tests/test_cli_config_helpers.py # tests/test_embed_batching.py
The chunking.min_chunk_size _PARAM_MAP entry had no corresponding Click option, leaving the knob TOML-only. Add the flag to index and watch mirroring --max-chunk-size, and document the key in the CLI/config/API references, including the caveat that enabling merging changes chunk IDs and triggers a one-time re-embed of affected files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
chunking.min_chunk_size: consecutive sections below the threshold are accumulated into one chunk (never exceedingmax_chunk_size), carrying the first section's heading and the full merged line span.0= exactly today's behavior (every section flushes immediately); the merge path only activates when opted in. Oversized sections still route through the existing paragraph splitter unchanged.ChunkingConfig.min_chunk_size(TOML +config set), allowed in project-local.memsearch.toml(same trust tier as the existingchunking.max_chunk_size/overlap_lines), passed throughMemSearch.__init__and the CLI override map.Test plan
max_chunk_size; pending chunks flush before an oversized section; accumulation stops once the threshold is reached_cfg_to_memsearch_kwargstranslation test and manual-construction test stubs for the new attributeruff check/ruff format --checkcleanDesign notes
###tail can merge into the following##section). For the transcript shape that motivates this feature, levels carry no hierarchy (every turn is the same level), and stopping at level boundaries would reintroduce the tiny-chunk problem exactly where it bites. If level-aware merging turns out to matter for prose docs, it can be layered on later without changing the config surface.content(sections include their heading lines), so no text is lost to search; the metadata heading is used for display/filtering and the first section's heading is the least surprising representative for a run of consecutive tiny sections.