Skip to content

Commit f819f12

Browse files
committed
Minor performance improvement
1 parent 55910cb commit f819f12

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

src/semchunk/semchunk.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,8 @@ def chunk(text: str, chunk_size: int, token_counter: callable, _recursion_depth:
9393

9494
# If the splitter is not whitespace and the split is not the last split, add the splitter to the end of the last chunk if doing so would not cause it to exceed the chunk size otherwise add the splitter as a new chunk.
9595
if not splitter_is_whitespace and not (i == len(splits) - 1 or all(j in skips for j in range(i+1, len(splits)))):
96-
if token_counter(chunks[-1]+splitter) <= chunk_size:
97-
chunks[-1] += splitter
96+
if token_counter(last_chunk_with_splitter:=chunks[-1]+splitter) <= chunk_size:
97+
chunks[-1] = last_chunk_with_splitter
9898

9999
else:
100100
chunks.append(splitter)

0 commit comments

Comments
 (0)