Commit beaef8b
feat: add UTF-8 CountTokens, parallel encode, and span-optimized encode paths
Add CountTokens(ReadOnlySpan<byte>) for zero-alloc token counting from
UTF-8 bytes using stackalloc/ArrayPool. Add EncodeParallel for large text
encoding via parallel BPE processing. Optimize Encode(ReadOnlySpan<char>)
and EncodeWithAllDisallowedSpecial with dedicated span-based code path
using NET9 alternate dictionary lookups. Update benchmark results and add
CountTokensUtf8 and EncodeParallel benchmark categories.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>1 parent 66e6894 commit beaef8b
File tree
5 files changed
+448
-178
lines changed- benchmarks
- src
- benchmarks/Tiktoken.Benchmarks
- libs/Tiktoken.Core
5 files changed
+448
-178
lines changed
0 commit comments