Improved chunking performance.

Author: umarbutler

CHANGELOG.md

@@ -1,8 +1,13 @@
 ## Changelog 🔄
 All notable changes to `semchunk` will be documented here. This project adheres to [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.1.1] - 2023-11-07
+### Changed
+- Improved chunking performance.
+
 ## [0.1.0] - 2023-11-05
 ### Added
 - Added the `chunk()` function, which splits text into semantically meaningful chunks of a specified size as determined by a provided token counter.
 
+[0.1.1]: https://github.com/umarbutler/semchunk/compare/v0.1.0...v0.1.1
 [0.1.0]: https://github.com/umarbutler/semchunk/releases/tag/v0.1.0

src/semchunk/semchunk.py

@@ -48,10 +48,6 @@ def chunk(text: str, chunk_size: int, token_counter: callable, _recursion_depth:
     
     Returns:
         list[str]: A list of chunks up to `chunk_size`-tokens-long, with any whitespace used to split the text removed."""
-    
-    # If the text is already within the chunk size, return it as the only chunk.
-    if token_counter(text) <= chunk_size:
-        return [text]
 
     # Split the text using the most semantically meaningful splitter possible.
     splitter, splits = _split_text(text)