Skip to content

Commit 43eab43

Browse files
authored
fix: fix typing issue (#162)
1 parent 6b18e20 commit 43eab43

File tree

1 file changed

+6
-5
lines changed

1 file changed

+6
-5
lines changed

model2vec/model.py

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -114,13 +114,14 @@ def save_pretrained(self, path: PathLike, model_name: str | None = None) -> None
114114
model_name=model_name,
115115
)
116116

117-
def tokenize(self, sentences: list[str], max_length: int | None = None) -> list[int]:
117+
def tokenize(self, sentences: list[str], max_length: int | None = None) -> list[list[int]]:
118118
"""
119-
Tokenize a sentence.
119+
Tokenize a list of sentences.
120120
121-
:param sentences: The sentence to tokenize.
122-
:param max_length: The maximum length of the sentence.
123-
:return: The tokens.
121+
:param sentences: The sentences to tokenize.
122+
:param max_length: The maximum length of the sentences in tokens. If this is None, sequences
123+
are not truncated.
124+
:return: A list of list of tokens.
124125
"""
125126
if max_length is not None:
126127
m = max_length * self.median_token_length

0 commit comments

Comments
 (0)