Skip to content

Commit beaef8b

Browse files
HavenDVclaude
andcommitted
feat: add UTF-8 CountTokens, parallel encode, and span-optimized encode paths
Add CountTokens(ReadOnlySpan<byte>) for zero-alloc token counting from UTF-8 bytes using stackalloc/ArrayPool. Add EncodeParallel for large text encoding via parallel BPE processing. Optimize Encode(ReadOnlySpan<char>) and EncodeWithAllDisallowedSpecial with dedicated span-based code path using NET9 alternate dictionary lookups. Update benchmark results and add CountTokensUtf8 and EncodeParallel benchmark categories. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 66e6894 commit beaef8b

File tree

5 files changed

+448
-178
lines changed

5 files changed

+448
-178
lines changed

0 commit comments

Comments
 (0)