-
Notifications
You must be signed in to change notification settings - Fork 9
Open
Description
Thinking about including a tokenizer class in the project.
I'm thinking the API could look like:
from iranlowo.tokenizer import Tokenizer
text = "some text"
word_tokens = Tokenizer(text).word_tokenize()
sentence_tokens = Tokenizer(text).sentence_tokenize()Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels