Skip to content

Switch to PyTorch TokenizersΒ #694

@IgorSwat

Description

@IgorSwat

ExecuTorch 1.0 comes with its own tokenizers implementation based on PyTorch Tokenizers. Since it is already being bundled into ExecuTorch binaries, we could get rid of tokenizers-cpp package and reduce the bundle size of our library.

Metadata

Metadata

Assignees

Labels

executorchIssues and tasks related specifically to ExecuTorchimportant ❗This issue has high priorityimprovementPRs or issues focused on improvements in the current codebase

Type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions