Commit bf4b32b
committed
Update extension/llm/tokenizers to d710a0cf10cfa8cb7ffda33c4e61af63119bc95f
1 parent a728efa commit bf4b32b
1 file changed
+1
-1
lines changedSubmodule tokenizers updated 13 files
- .github/workflows/pull.yml+2-2
- TARGETS+5
- cmake/tokenizers-config.cmake.in+5-1
- examples/tokenize_tool/main.cpp+1-1
- include/pytorch/tokenizers/compiler.h+4-2
- include/pytorch/tokenizers/tekken.h+1
- src/hf_tokenizer.cpp+9-9
- src/pre_tokenizer.cpp+8-6
- targets.bzl+24
- test/fb/TARGETS-30
- test/test_hf_tokenizer.py+13
- third-party/TARGETS+5
- third-party/targets.bzl+3
0 commit comments