[ML] Add DeBERTa-V2/V3 tokenizer#111852

Merged

maxhniebergall merged 25 commits intomainfrom

addDebertaTokenizer

Oct 3, 2024

Commits on Sep 17, 2024

start to add deberta-v2 tokenizer classes
Max Hniebergall
committed
continue to add basic tokenizer stuff
Max Hniebergall
committed
Finish adding DeBERTa-2 tokenizer
Max Hniebergall
committed
Complete test setup and linting
Max Hniebergall
committed
Update docs/changelog/111852.yaml

maxhniebergall
authored and
Max Hniebergall
committed
Add serialization of deberta tokenization
Max Hniebergall
committed
fix request buillder to match model
Max Hniebergall
committed
debugging
Max Hniebergall
committed
add balanced truncation
Max Hniebergall
committed
remove full vocabulary and use tiny vocab for tests
Max Hniebergall
committed
Remove TODO
Max Hniebergall
committed
precommit
Max Hniebergall
committed
Add named writables and known tokenizers
Max Hniebergall
committed
Add deberta to list of known tokenizers in test
Max Hniebergall
committed
Add tests for balanced tokenizer and fix errors in tokenizer logic
Max Hniebergall
committed
fix order of parameters passed to deberta
Max Hniebergall
committed
Add support for byte_fallback which is enabled for DeBERTa
Max Hniebergall
committed
precommit
Max Hniebergall
committed

Commits on Sep 18, 2024

Merge branch 'main' into addDebertaTokenizer
elasticmachine
authored
update tests to account for byte decomposition
Max Hniebergall
committed
remove sysout
Max Hniebergall
committed
fix tests for byteFallback, for real this time
Max Hniebergall
committed

Commits on Oct 3, 2024

Move defaultSpanForChunking into super class to avoid repitition
Max Hniebergall
committed
simplify decomposeBytePieces
Max Hniebergall
committed
Merge branch 'main' of github.com:elastic/elasticsearch into addDebertaTokenizer
Max Hniebergall
committed