File tree Expand file tree Collapse file tree 1 file changed +6
-6
lines changed
Expand file tree Collapse file tree 1 file changed +6
-6
lines changed Original file line number Diff line number Diff line change @@ -76,12 +76,12 @@ All tokenizers in the library implement the <xref:Microsoft.ML.Tokenizers.Tokeni
7676
7777| Method | Description |
7878| -------------------------------------------------------| --------------------------------------|
79- | < xref:Microsoft.ML.Tokenizers.Tokenizer.EncodeToIds* > | Converts text to a list of token IDs |
80- | < xref:Microsoft.ML.Tokenizers.Tokenizer.Decode* > | Converts token IDs back to text |
81- | < xref:Microsoft.ML.Tokenizers.Tokenizer.CountTokens* > | Returns the number of tokens in a text string |
82- | < xref:Microsoft.ML.Tokenizers.Tokenizer.EncodeToTokens* > | Returns detailed token information including values and IDs |
83- | < xref:Microsoft.ML.Tokenizers.Tokenizer.GetIndexByTokenCount* > | Finds the character index for a specific token count from the start |
84- | < xref:Microsoft.ML.Tokenizers.Tokenizer.GetIndexByTokenCountFromEnd* > | Finds the character index for a specific token count from the end |
79+ | < xref:Microsoft.ML.Tokenizers.Tokenizer.EncodeToIds* > | Converts text to a list of token IDs. |
80+ | < xref:Microsoft.ML.Tokenizers.Tokenizer.Decode* > | Converts token IDs back to text. |
81+ | < xref:Microsoft.ML.Tokenizers.Tokenizer.CountTokens* > | Returns the number of tokens in a text string. |
82+ | < xref:Microsoft.ML.Tokenizers.Tokenizer.EncodeToTokens* > | Returns detailed token information including values and IDs. |
83+ | < xref:Microsoft.ML.Tokenizers.Tokenizer.GetIndexByTokenCount* > | Finds the character index for a specific token count from the start. |
84+ | < xref:Microsoft.ML.Tokenizers.Tokenizer.GetIndexByTokenCountFromEnd* > | Finds the character index for a specific token count from the end. |
8585
8686## Migrate from other libraries
8787
You can’t perform that action at this time.
0 commit comments