feat: add fasttokens benchmarks and tokenizer backend docs#7388
Open
biswapanda wants to merge 3 commits intobis/fast-tokens-dynamofrom
Open
feat: add fasttokens benchmarks and tokenizer backend docs#7388biswapanda wants to merge 3 commits intobis/fast-tokens-dynamofrom
biswapanda wants to merge 3 commits intobis/fast-tokens-dynamofrom
Conversation
github-actions
bot
added
feat
documentation
Contributor
biswapanda
requested review from
dagil-nvidia and
nealvaidya
and removed request for
a team
nealvaidya
approved these changes
Mar 14, 2026
Contributor
nealvaidya
left a comment
nealvaidya
left a comment
There was a problem hiding this comment.
Approving from docs standpoint
biswapanda
force-pushed
the
bis/fast-tokens-bench-docs
branch
from
24f4714 to
f8e8366
Compare
Benchmarks: - Rename benches/tokenizer.rs to benches/tokenizer_simple.rs, add criterion fasttokens vs HF encode and batch-encode benchmarks - Add benches/tokenizer_dataset.rs: dataset-driven benchmark using LongBench-v2 (503 real-world samples), sequential and batched modes with correctness verification (~24x sequential, ~27x batched speedup) Docs: - docs/components/frontend/tokenizer-backends.md: user guide with configuration, compatibility notes, and benchmark results - docs/components/frontend/configuration.md: added Tokenizer section - docs/index.yml: added Tokenizer Backends page under Frontend
biswapanda
force-pushed
the
bis/fast-tokens-bench-docs
branch
from
49e8a7d to
34eddf7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview:
Benchmarks:
Docs:
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)