Skip to content

Commit 1305237

Browse files
gewarrentarekgh
andauthored
Apply suggestions from code review
Co-authored-by: Tarek Mahmoud Sayed <[email protected]>
1 parent f29a826 commit 1305237

File tree

1 file changed

+5
-7
lines changed

1 file changed

+5
-7
lines changed

docs/core/whats-new/dotnet-9/overview.md

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -66,14 +66,12 @@ The [Microsoft.ML.Tokenizers](https://www.nuget.org/packages/Microsoft.ML.Tokeni
6666
The latest release introduces significant new capabilities:
6767

6868
- Tokenizers
69-
- SentencePiece
69+
- Tiktoken for GPT (3, 3.5, 4, 4o, o1) and Llam3 models
70+
- Llama (based on SentencePiece) for Llama and Mistral models
71+
- CodeGen for code generation models like codegen-350M-mono
72+
- Phi2 (based on CodeGen) for Microsoft Phi2 model
7073
- WordPiece
71-
- Built-in tokenizers for the following models:
72-
- GPT (3, 3.5, 4, 4o, o1)
73-
- Llama
74-
- Phi
75-
- BERT
76-
- CodeGen
74+
- Bert (based on WordPiece) for Bert supported models like optimum--all-MiniLM-L6-v2
7775

7876
### Tensors
7977

0 commit comments

Comments
 (0)