Reduce crate size

[ArthurZucker]

Reduce tokenizers crate size

Reduce the on-device library size of the tokenizers crate from 2.65 MB → 0.96 MB (64% reduction) for inference-only deployments.

Size reduction breakdown

Measured on macOS arm64, stripped cdylib, LTO fat, opt-level=s:

main (today, all features)                                     2.65 MB
├── drop deps (derive_builder, itertools, monostate, ahash)   -0.32 MB  (done)
├── feature-gate training/spm/parallel/unicode-norm           -0.34 MB  (done)
├── make regex optional                                       -0.65 MB  (done)
├── drop dary_heap serde feature                              -tiny     (done)
├── codegen-units = 1                                         -0.03 MB  (done, in profile)
├── panic = "abort"                                        \
├── -Zlocation-detail=none -Zfmt-debug=none                 } -0.35 MB combined
├── build-std=std,panic_abort optimize_for_size            /
└───────────────────────────────────────────────
    TOTAL with nightly build-std                               0.96 MB ✓
    TOTAL with stable (codegen-units=1, panic=abort)          ~1.18 MB

Comparison with Meta pytorch/tokenizers (C++)

	Meta (C++)	HuggingFace (Rust, this PR)
Stripped binary	~0.8 MB	0.96 MB (nightly) / 1.18 MB (stable)
Models	BPE, SentencePiece, Tiktoken	BPE, WordPiece, Unigram, WordLevel
Normalizers	Replace, Prepend, NFC, Sequence	All of the above + NFKC/NFD/NFKD, Bert, Lowercase, Strip, StripAccents, ByteLevel, NMT
Pre-tokenizers	Regex, Digits, ByteLevel, Sequence	All of the above + Whitespace, Metaspace, Punctuation, Split, UnicodeScripts, BertPreTokenizer, CharDelimiter, FixedLength
Post-processors	TemplateProcessing, Sequence	All of the above + BertProcessing, RobertaProcessing, ByteLevel
Decoders	Basic token decoder	BPE, ByteLevel, WordPiece, Metaspace, CTC, ByteFallback, Fuse, Strip, Replace, Sequence
Added tokens	Basic special tokens only	Full AddedToken with single-word, lstrip/rstrip, normalized, per-token config
Training	❌	✅ (feature-gated)
Padding / Truncation	❌	✅
Batch encoding	❌	✅ (with optional rayon parallelism)
tokenizer.json loading	✅ (partial — many post-processors are TODO)	✅ (full)

We're ~20% larger than Meta's C++ implementation while supporting significantly more features. The gap is primarily serde's JSON deserialization infrastructure (~225 KB) which Meta avoids by using hardcoded loaders.

What changed

Dependency cleanup:

• Replaced derive_builder with manual builders
• Replaced itertools with std equivalents
• Replaced ahash with foldhash (smaller, faster on benchmarks)
• Dropped monostate
• Removed serde feature from dary_heap (never serialized)

Feature gates (all backward-compatible, enabled by default):

• training — gates rand, esaxx-rs, compact_str, all trainer impls
• spm — gates spm_precompiled, unicode-segmentation
• parallel — gates rayon, rayon-cond
• unicode-normalization — gates NFC/NFD/NFKC/NFKD normalizers
• regex — new: gates the Rust regex crate. When disabled, all regex operations use the system regex engine (onig or fancy-regex). Replaced 4 regex statics in added_vocabulary.rs with char ops, Whitespace pre-tokenizer with SysRegex, Pattern for &str with str::match_indices.

Build profiles:

• Added release-small profile with opt-level = "s", strip = true, panic = "abort", codegen-units = 1
• Documented nightly build-std command for sub-1MB builds

CI:

• Added bundle size reporting via $GITHUB_STEP_SUMMARY to Rust and Python release workflows
• Fixed macOS abi3 cross-compilation RUSTFLAGS in CI

Minimal inference-only configuration

# Cargo.toml — smallest possible, uses Oniguruma regex (C dep)
tokenizers = { version = "0.22", default-features = false, features = ["onig"] }

# Or pure Rust for WASM (no C dependencies)
tokenizers = { version = "0.22", default-features = false, features = ["unstable_wasm"] }

For the absolute smallest binary (nightly):

RUSTFLAGS="-Zlocation-detail=none -Zfmt-debug=none" cargo +nightly build \
  -Z build-std=std,panic_abort -Z build-std-features="optimize_for_size" \
  --target aarch64-apple-darwin --profile release-small

Benchmark results (ahash → foldhash)

Foldhash is equal-or-faster on all benchmarks. Training is 7-10% faster, encoding is 2-3% faster.

Benchmark	ahash (main)	foldhash (this PR)	Delta
BPE GPT2 encode	1.718 s	1.668 s	-3.0%
BPE train large	958 ms	883 ms	-7.8%
llama3 encode	1.687 s	1.644 s	-2.6%
llama3 train big	1.124 s	1.016 s	-9.6%
unigram train big	675 ms	614 ms	-9.1%
BERT encode	1.560 s	1.558 s	-0.1%
BERT train big	901 ms	871 ms	-3.3%

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

/benchmark

[ArthurZucker]

/benchmark

[ArthurZucker]

/benchmark

[github-actions]

Python Benchmark Results

Commit: 994022f1f942484156d7b339c27e6bfebbdacb18

[github-actions]

Rust Benchmark Results

Commit: 994022f1f942484156d7b339c27e6bfebbdacb18