Skip to content

Commit 3c2d8a0

Browse files
committed
Fixed typos in README.
1 parent 5491fdd commit 3c2d8a0

File tree

2 files changed

+4
-1
lines changed

2 files changed

+4
-1
lines changed

CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ All notable changes to `semchunk` will be documented here. This project adheres
55
### Added
66
- Memoized `chunk()`.
77

8+
### Fixed
9+
- Fixed typos in README.
10+
811
## [0.2.0] - 2023-11-07
912
### Added
1013
- Added the `memoize` argument to `chunk()`, which memoizes token counters by default to significantly improve performance.

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ The code snippet below demonstrates how text can be chunked with `semchunk`:
1919
>>> text = 'The quick brown fox jumps over the lazy dog.'
2020
>>> chunk_size = 2 # A low chunk size is used here for demo purposes.
2121
>>> encoder = tiktoken.encoding_for_model('gpt-4')
22-
>>> token_counter = lambda text: len(tiktoken.encoding_for_model(model).encode(text)) # `token_counter` may be swapped out for any function capable of counting tokens.
22+
>>> token_counter = lambda text: len(encoder.encode(text)) # `token_counter` may be swapped out for any function capable of counting tokens.
2323
>>> semchunk.chunk(text, chunk_size=chunk_size, token_counter=token_counter)
2424
['The quick', 'brown fox', 'jumps over', 'the lazy', 'dog.']
2525
```

0 commit comments

Comments
 (0)