Reorganize benchmark to include fairer comparisons#27

Merged

hendrikvanantwerpen merged 14 commits intomainfrom

add-input-splitting

Oct 14, 2024

Commits on Oct 9, 2024

Add Huggingface benchmark
Hendrik van Antwerpen
committed
Use pretrained model for huggingface tokenizer and add equivalence test
Hendrik van Antwerpen
committed
Support input splitting for openai tokenizers
Hendrik van Antwerpen
committed
Add huggingface without pre-tokenization
Hendrik van Antwerpen
committed
Update benchmark results and text
Hendrik van Antwerpen
committed
Add remark about performance impact of pre-tokenization
Hendrik van Antwerpen
committed
lint
Hendrik van Antwerpen
committed

Commits on Oct 10, 2024

README tweaks
Hendrik van Antwerpen
committed
Add doc comment to type
Hendrik van Antwerpen
committed
Fix incorrect benchmark text
Hendrik van Antwerpen
committed
Update text about input splitting
Hendrik van Antwerpen
committed
fmt
Hendrik van Antwerpen
committed

Commits on Oct 14, 2024

Fix typo

Hendrik van Antwerpen
and
aneubeck
authored
section -> substring
Hendrik van Antwerpen
committed