Commit 814b4c0
feat: add DecodeToUtf8 zero-alloc API, o200k benchmarks, and optimize Explore allocations
- Add DecodeToUtf8/GetDecodedUtf8ByteCount span API for zero-allocation decode into caller buffers
- Add o200k_base benchmark comparisons (Encode, CountTokens, Decode vs cl100k)
- Optimize Explore/ExploreUtfSafe with stackalloc to reduce allocations by 23%
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>1 parent a2daddb commit 814b4c0
File tree
5 files changed
+349
-138
lines changed- benchmarks
- src
- benchmarks/Tiktoken.Benchmarks
- libs/Tiktoken.Core
5 files changed
+349
-138
lines changed
0 commit comments