Skip to content

Commit cd51406

Browse files
gewarrenCopilot
andauthored
Update docs/ai/how-to/use-tokenizers.md
Co-authored-by: Copilot <[email protected]>
1 parent 9be4112 commit cd51406

File tree

1 file changed

+6
-6
lines changed

1 file changed

+6
-6
lines changed

docs/ai/how-to/use-tokenizers.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -76,12 +76,12 @@ All tokenizers in the library implement the <xref:Microsoft.ML.Tokenizers.Tokeni
7676

7777
| Method | Description |
7878
|-------------------------------------------------------|--------------------------------------|
79-
| <xref:Microsoft.ML.Tokenizers.Tokenizer.EncodeToIds*> | Converts text to a list of token IDs |
80-
| <xref:Microsoft.ML.Tokenizers.Tokenizer.Decode*> | Converts token IDs back to text |
81-
| <xref:Microsoft.ML.Tokenizers.Tokenizer.CountTokens*> | Returns the number of tokens in a text string |
82-
| <xref:Microsoft.ML.Tokenizers.Tokenizer.EncodeToTokens*> | Returns detailed token information including values and IDs |
83-
| <xref:Microsoft.ML.Tokenizers.Tokenizer.GetIndexByTokenCount*> | Finds the character index for a specific token count from the start |
84-
| <xref:Microsoft.ML.Tokenizers.Tokenizer.GetIndexByTokenCountFromEnd*> | Finds the character index for a specific token count from the end |
79+
| <xref:Microsoft.ML.Tokenizers.Tokenizer.EncodeToIds*> | Converts text to a list of token IDs. |
80+
| <xref:Microsoft.ML.Tokenizers.Tokenizer.Decode*> | Converts token IDs back to text. |
81+
| <xref:Microsoft.ML.Tokenizers.Tokenizer.CountTokens*> | Returns the number of tokens in a text string. |
82+
| <xref:Microsoft.ML.Tokenizers.Tokenizer.EncodeToTokens*> | Returns detailed token information including values and IDs. |
83+
| <xref:Microsoft.ML.Tokenizers.Tokenizer.GetIndexByTokenCount*> | Finds the character index for a specific token count from the start. |
84+
| <xref:Microsoft.ML.Tokenizers.Tokenizer.GetIndexByTokenCountFromEnd*> | Finds the character index for a specific token count from the end. |
8585

8686
## Migrate from other libraries
8787

0 commit comments

Comments
 (0)