Skip to content

Commit 5781bb4

Browse files
committed
Updated benchmarks.
1 parent 570195c commit 5781bb4

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ To ensure that chunks are as semantically meaningful as possible, `semchunk` use
103103
`semchunk` also relies on memoization to cache the results of token counters and the `chunk()` function, thereby improving performance.
104104

105105
## Benchmarks 📊
106-
On a desktop with a Ryzen 3600, 64 GB of RAM, Windows 11 and Python 3.11.4, it takes `semchunk` 8.34 seconds to split every sample in [NLTK's Gutenberg Corpus](https://www.nltk.org/howto/corpus.html#plaintext-corpora) into 512-token-long chunks with GPT-4's tokenizer (for context, the Corpus contains 18 texts and 3,001,260 tokens). By comparison, it takes [`semantic-text-splitter`](https://pypi.org/project/semantic-text-splitter/) 116.59 seconds to chunk the same texts into 512-token-long chunks — a difference of 92.84%.
106+
On a desktop with a Ryzen 3600, 64 GB of RAM, Windows 11 and Python 3.11.9, it takes `semchunk` 6.69 seconds to split every sample in [NLTK's Gutenberg Corpus](https://www.nltk.org/howto/corpus.html#plaintext-corpora) into 512-token-long chunks with GPT-4's tokenizer (for context, the Corpus contains 18 texts and 3,001,260 tokens). By comparison, it takes [`semantic-text-splitter`](https://pypi.org/project/semantic-text-splitter/) 116.48 seconds to chunk the same texts into 512-token-long chunks — a difference of 94.26%.
107107

108108
The code used to benchmark `semchunk` and `semantic-text-splitter` is available [here](https://github.com/umarbutler/semchunk/blob/main/tests/bench.py).
109109

0 commit comments

Comments
 (0)