ML.NET to AI and Machine Learning

docs/core/whats-new/dotnet-9/overview.md

@@ -53,9 +53,59 @@ The .NET 9 SDK introduces _workload sets_, where all of your workloads stay at a
 
 For more information, see [What's new in the SDK for .NET 9](sdk.md).
 
+## AI building blocks and fundamentals
+
+### Microsoft.Extensions.AI & Microsoft.Extensions.VectorData
+
+.NET 9 introduces a unified layer of C# abstractions through the [Microsoft.Extensions.AI](https://www.nuget.org/packages/Microsoft.Extensions.AI.Abstractions/) and [Microsoft.Extensions.VectorData](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions/) packages. These abstractions facilitate interaction with AI services, including small and large language models (SLMs and LLMs), embeddings, vector stores, and middleware.
+
+### Tokenizers
+
+The [Microsoft.ML.Tokenizers](https://www.nuget.org/packages/Microsoft.ML.Tokenizers) library provides .NET developers with capabilities for encoding and decoding text to tokens. For AI scenarios, this is important to manage context, calculate cost, and preprocess text when working with local models.
+
+The latest release introduces significant new capabilities:
+
+- Tokenizers
+  - SentencePiece
+  - WordPiece
+- Built-in tokenizers for the following models:
+  - GPT (3, 3.5, 4, 4o, o1)
+  - Llama
+  - Phi
+  - BERT
+  - CodeGen
+
+### Tensors
+
+In .NET 9, `TensorPrimitives` and the new `Tensor<T>` type expand AI capabilities by enabling efficient encoding, manipulation, and computation of multi-dimensional data.
+
+Improvements in the latest release of [System.Numerics.Tensors](https://www.nuget.org/packages/System.Numerics.Tensors/) include:
+
+#### TensorPrimitives
+
+- **Expanded Method Scope:** Increased from 40 to nearly 200 overloads, now including numerical operations similar to `Math`, `MathF`, and `INumber<T>`, but for spans of values.
+- **Performance Enhancements:** Many operations are now SIMD-optimized for better performance.
+- **Generic Overloads:** Supports any T that implements a certain interface, expanding beyond just spans of float values in .NET.
+
+#### Tensor
+
+- Builds on top of `TensorPrimitives` for efficient math operations.
+- Provides efficient interop with AI libraries (ML.NET, TorchSharp, ONNX Runtime) using zero copies where possible.
+- Enables easy and efficient data manipulation with indexing and slicing operations.
+
 ## ML.NET
 
-ML.NET is an open-source, cross-platform framework that enables integration of custom machine-learning models into .NET applications. The latest version, ML.NET 4.0, adds [additional tokenizer support](../../../machine-learning/whats-new/overview.md#additional-tokenizer-support) for tokenizers such as Tiktoken and models such as Llama and CodeGen. <!--Add info about `Tensor<T>` here and in what's new for ML.NET.-->
+[ML.NET](https://www.nuget.org/packages/Microsoft.ML/) is an open-source, cross-platform framework that enables integration of custom machine-learning models into .NET applications.
+
+ML.NET 4.0 brings the following improvements:
+
+- New ways to programatically configure `MLContext` options.
+- Load ONNX models as `Stream`.
+- DataFrame improvements.
+- (Experimental) TorchSharp ports of Llama and Phi family of models.
+- (Experimental) CausalLM pipeline APIs
+
+For more information, see [What's new in ML.NET](dotnet/machine-learning/whats-new/overview).
 
 ## .NET Aspire