move tokenizers under ml.net

Author: gewarren

docs/core/whats-new/dotnet-9/overview.md

@@ -53,58 +53,52 @@ The .NET 9 SDK introduces _workload sets_, where all of your workloads stay at a
 
 For more information, see [What's new in the SDK for .NET 9](sdk.md).
 
-## AI building blocks and fundamentals
-
-### Microsoft.Extensions.AI & Microsoft.Extensions.VectorData
+## AI building blocks
 
 .NET 9 introduces a unified layer of C# abstractions through the [Microsoft.Extensions.AI](https://www.nuget.org/packages/Microsoft.Extensions.AI.Abstractions/) and [Microsoft.Extensions.VectorData](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions/) packages. These abstractions facilitate interaction with AI services, including small and large language models (SLMs and LLMs), embeddings, vector stores, and middleware.
 
-### Tokenizers
-
-The [Microsoft.ML.Tokenizers](https://www.nuget.org/packages/Microsoft.ML.Tokenizers) library provides .NET developers with capabilities for encoding and decoding text to tokens. For AI scenarios, this is important to manage context, calculate cost, and preprocess text when working with local models.
-
-The latest release introduces significant new capabilities:
-
-- Tokenizers
-  - Tiktoken for GPT (3, 3.5, 4, 4o, o1) and Llam3 models
-  - Llama (based on SentencePiece) for Llama and Mistral models
-  - CodeGen for code generation models like codegen-350M-mono
-  - Phi2 (based on CodeGen) for Microsoft Phi2 model
-  - WordPiece
-  - Bert (based on WordPiece) for Bert supported models like optimum--all-MiniLM-L6-v2
-
-### Tensors
-
-In .NET 9, `TensorPrimitives` and the new `Tensor<T>` type expand AI capabilities by enabling efficient encoding, manipulation, and computation of multi-dimensional data.
+.NET 9 also includes new tensor types that expand AI capabilities. <xref:System.Numerics.Tensors.TensorPrimitives> and the new <xref:System.Numerics.Tensors.Tensor%601> type expand AI capabilities by enabling efficient encoding, manipulation, and computation of multi-dimensional data. You can find these types in the latest release of the [System.Numerics.Tensors package](https://www.nuget.org/packages/System.Numerics.Tensors/).
 
-Improvements in the latest release of [System.Numerics.Tensors](https://www.nuget.org/packages/System.Numerics.Tensors/) include:
+### TensorPrimitives
 
-#### TensorPrimitives
+- Expanded method scope: Increased from 40 to nearly 200 overloads, now including numerical operations similar to `Math`, `MathF`, and `INumber<T>` but for spans of values.
+- Performance enhancements: Many operations are now SIMD-optimized for better performance.
+- Generic overloads: Supports any type `T` that implements a certain interface, expanding beyond just spans of float values in .NET.
 
-- **Expanded Method Scope:** Increased from 40 to nearly 200 overloads, now including numerical operations similar to `Math`, `MathF`, and `INumber<T>`, but for spans of values.
-- **Performance Enhancements:** Many operations are now SIMD-optimized for better performance.
-- **Generic Overloads:** Supports any T that implements a certain interface, expanding beyond just spans of float values in .NET.
-
-#### Tensor
+### Tensor\<T>
 
 - Builds on top of `TensorPrimitives` for efficient math operations.
 - Provides efficient interop with AI libraries (ML.NET, TorchSharp, ONNX Runtime) using zero copies where possible.
 - Enables easy and efficient data manipulation with indexing and slicing operations.
 
-## ML.NET
+### ML.NET
 
 [ML.NET](https://www.nuget.org/packages/Microsoft.ML/) is an open-source, cross-platform framework that enables integration of custom machine-learning models into .NET applications.
 
 ML.NET 4.0 brings the following improvements:
 
-- New ways to programatically configure `MLContext` options.
+- New ways to programmatically configure `MLContext` options.
 - Load ONNX models as `Stream`.
 - DataFrame improvements.
+- New capabilities for [tokenizers](#tokenizers).
 - (Experimental) TorchSharp ports of Llama and Phi family of models.
-- (Experimental) CausalLM pipeline APIs
+- (Experimental) CausalLM pipeline APIs.
 
 For more information, see [What's new in ML.NET](../../../machine-learning/whats-new/overview.md).
 
+#### Tokenizers
+
+The [Microsoft.ML.Tokenizers](https://www.nuget.org/packages/Microsoft.ML.Tokenizers) library provides .NET developers with capabilities for encoding and decoding text to tokens. For AI scenarios, this is important to manage context, calculate cost, and preprocess text when working with local models.
+
+The latest release introduces significant new capabilities for tokenizers:
+
+- Tiktoken for GPT (3, 3.5, 4, 4o, o1) and Llam3 models
+- Llama (based on SentencePiece) for Llama and Mistral models
+- CodeGen for code-generation models like codegen-350M-mono
+- Phi2 (based on CodeGen) for Microsoft Phi2 model
+- WordPiece
+- Bert (based on WordPiece) for Bert-supported models like optimum--all-MiniLM-L6-v2
+
 ## .NET Aspire
 
 .NET Aspire is an opinionated, cloud-ready stack for building observable, production ready, distributed applications.​ .NET Aspire is delivered through a collection of NuGet packages that handle specific cloud-native concerns, and is available in preview for .NET 9. For more information, see [.NET Aspire](/dotnet/aspire).