Skip to content

Commit 2033d4a

Browse files
luisquintanillaadamsitnikgewarren
authored
Data Ingestion (#49051)
* Initial commit * first part of the doc update * add missing samples, update type names and some of the descriptions * address Copilot feedback * Apply suggestions from code review Co-authored-by: Genevieve Warren <[email protected]> * Apply suggestions from code review Co-authored-by: Genevieve Warren <[email protected]> * Updated image * Fix image naming * Update link to image * Update date * Fixing image rendering in dark mode --------- Co-authored-by: Adam Sitnik <[email protected]> Co-authored-by: Genevieve Warren <[email protected]>
1 parent 7e5308a commit 2033d4a

File tree

3 files changed

+161
-0
lines changed

3 files changed

+161
-0
lines changed

‎docs/ai/conceptual/data-ingestion.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
---
2+
title: Data ingestion
3+
description: Introduction to data ingestion
4+
author: luisquintanilla
5+
ms.author: luquinta
6+
ms.date: 12/02/2025
7+
ms.topic: concept-article
8+
ai-usage: ai-assisted
9+
---
10+
11+
# Data ingestion
12+
13+
Data ingestion is the process of collecting, reading, and preparing data from different sources such as files, databases, APIs, or cloud services so it can be used in downstream applications. In practice, this process follows the Extract-Transform-Load (ETL) workflow:
14+
15+
- **Extract** data from its original source, whether that is a PDF, Word document, audio file, or web API.
16+
- **Transform** the data by cleaning, chunking, enriching, or converting formats.
17+
- **Load** the data into a destination like a database, vector store, or AI model for retrieval and analysis.
18+
19+
For AI and machine learning scenarios, especially Retrieval-Augmented Generation (RAG), data ingestion is not just about converting data from one format to another. It is about making data usable for intelligent applications. This means representing documents in a way that preserves their structure and meaning, splitting them into manageable chunks, enriching them with metadata or embeddings, and storing them so they can be retrieved quickly and accurately.
20+
21+
## Why data ingestion matters for AI applications
22+
23+
Imagine you're building a RAG-powered chatbot to help employees find information across your company's vast collection of documents. These documents might include PDFs, Word files, PowerPoint presentations, and web pages scattered across different systems.
24+
25+
Your chatbot needs to understand and search through thousands of documents to provide accurate, contextual answers. But raw documents aren't suitable for AI systems. You need to transform them into a format that preserves meaning while making them searchable and retrievable.
26+
27+
This is where data ingestion becomes critical. You need to extract text from different file formats, break large documents into smaller chunks that fit within AI model limits, enrich the content with metadata, generate embeddings for semantic search, and store everything in a way that enables fast retrieval. Each step requires careful consideration of how to preserve the original meaning and context.
28+
29+
## The Microsoft.Extensions.DataIngestion library
30+
31+
The [📦 Microsoft.Extensions.DataIngestion package](https://www.nuget.org/packages/Microsoft.Extensions.DataIngestion) provides foundational .NET building blocks for data ingestion. It enables developers to read, process, and prepare documents for AI and machine learning workflows, especially Retrieval-Augmented Generation (RAG) scenarios.
32+
33+
With these building blocks, you can create robust, flexible, and intelligent data ingestion pipelines tailored for your application needs:
34+
35+
- **Unified document representation:** Represent any file type (for example, PDF, Image, or Microsoft Word) in a consistent format that works well with large language models.
36+
- **Flexible data ingestion:** Read documents from both cloud services and local sources using multiple built-in readers, making it easy to bring in data from wherever it lives.
37+
- **Built-in AI enhancements:** Automatically enrich content with summaries, sentiment analysis, keyword extraction, and classification, preparing your data for intelligent workflows.
38+
- **Customizable chunking strategies:** Split documents into chunks using token-based, section-based, or semantic-aware approaches, so you can optimize for your retrieval and analysis needs.
39+
- **Production-ready storage:** Store processed chunks in popular vector databases and document stores, with support for embedding generation, making your pipelines ready for real-world scenarios.
40+
- **End-to-end pipeline composition:** Chain together readers, processors, chunkers, and writers with the <xref:Microsoft.Extensions.DataIngestion.IngestionPipeline`1> API, reducing boilerplate and making it easy to build, customize, and extend complete workflows.
41+
- **Performance and scalability:** Designed for scalable data processing, these components can handle large volumes of data efficiently, making them suitable for enterprise-grade applications.
42+
43+
All of these components are open and extensible by design. You can add custom logic and new connectors, and extend the system to support emerging AI scenarios. By standardizing how documents are represented, processed, and stored, .NET developers can build reliable, scalable, and maintainable data pipelines without "reinventing the wheel" for every project.
44+
45+
### Built on stable foundations
46+
47+
![Data Ingestion Architecture Diagram](../media/data-ingestion/dataingestion.png)
48+
49+
These data ingestion building blocks are built on top of proven and extensible components in the .NET ecosystem, ensuring reliability, interoperability, and seamless integration with existing AI workflows:
50+
51+
- **Microsoft.ML.Tokenizers:** Tokenizers provide the foundation for chunking documents based on tokens. This enables precise splitting of content, which is essential for preparing data for large language models and optimizing retrieval strategies.
52+
- **Microsoft.Extensions.AI:** This set of libraries powers enrichment transformations using large language models. It enables features like summarization, sentiment analysis, keyword extraction, and embedding generation, making it easy to enhance your data with intelligent insights.
53+
- **Microsoft.Extensions.VectorData:** This set of libraries offers a consistent interface for storing processed chunks in a wide variety of vector stores, including Qdrant, Azure SQL, CosmosDB, MongoDB, ElasticSearch, and many more. This ensures your data pipelines are ready for production and can scale across different storage backends.
54+
55+
In addition to familiar patterns and tools, these abstractions build on already extensible components. Plug-in capability and interoperability are paramount, so as the rest of the .NET AI ecosystem grows, the capabilities of the data ingestion components grow as well. This approach empowers developers to easily integrate new connectors, enrichments, and storage options, keeping their pipelines future-ready and adaptable to evolving AI scenarios.
56+
57+
## Data ingestion building blocks
58+
59+
The [Microsoft.Extensions.DataIngestion](https://www.nuget.org/packages/Microsoft.Extensions.DataIngestion) library is built around several key components that work together to create a complete data processing pipeline. This section explores each component and how they fit together.
60+
61+
### Documents and document readers
62+
63+
At the foundation of the library is the <xref:Microsoft.Extensions.DataIngestion.IngestionDocument> type, which provides a unified way to represent any file format without losing important information. `IngestionDocument` is Markdown-centric because large language models work best with Markdown formatting.
64+
65+
The <xref:Microsoft.Extensions.DataIngestion.IngestionDocumentReader> abstraction handles loading documents from various sources, whether local files or streams. A few readers are available:
66+
67+
- **[MarkItDown](https://www.nuget.org/packages/Microsoft.Extensions.DataIngestion.MarkItDown)**
68+
- **[Markdig](https://www.nuget.org/packages/Microsoft.Extensions.DataIngestion.Markdig/)**
69+
70+
More readers (including **LlamaParse** and **Azure Document Intelligence**) will be added in the future.
71+
72+
This design means you can work with documents from different sources using the same consistent API, making your code more maintainable and flexible.
73+
74+
### Document processing
75+
76+
Document processors apply transformations at the document level to enhance and prepare content. The library provides the <xref:Microsoft.Extensions.DataIngestion.ImageAlternativeTextEnricher> class as a built-in processor that uses large language models to generate descriptive alternative text for images within documents.
77+
78+
### Chunks and chunking strategies
79+
80+
Once you have a document loaded, you typically need to break it down into smaller pieces called chunks. Chunks represent subsections of a document that can be efficiently processed, stored, and retrieved by AI systems. This chunking process is essential for retrieval-augmented generation scenarios where you need to find the most relevant pieces of information quickly.
81+
82+
The library provides several chunking strategies to fit different use cases:
83+
84+
- **Header-based chunking** to split on headers.
85+
- **Section-based chunking** to split on sections (for example, pages).
86+
- **Semantic-aware chunking** to preserve complete thoughts.
87+
88+
These chunking strategies build on the Microsoft.ML.Tokenizers library to intelligently split text into appropriately sized pieces that work well with large language models. The right chunking strategy depends on your document types and how you plan to retrieve information.
89+
90+
```csharp
91+
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4");
92+
IngestionChunkerOptions options = new(tokenizer)
93+
{
94+
MaxTokensPerChunk = 2000,
95+
OverlapTokens = 0
96+
};
97+
IngestionChunker<string> chunker = new HeaderChunker(options);
98+
```
99+
100+
### Chunk processing and enrichment
101+
102+
After documents are split into chunks, you can apply processors to enhance and enrich the content. Chunk processors work on individual pieces and can perform:
103+
104+
- **Content enrichment** including automatic summaries (`SummaryEnricher`), sentiment analysis (`SentimentEnricher`), and keyword extraction (`KeywordEnricher`).
105+
- **Classification** for automated content categorization based on predefined categories (`ClassificationEnricher`).
106+
107+
These processors use [Microsoft.Extensions.AI.Abstractions](https://www.nuget.org/packages/Microsoft.Extensions.AI.Abstractions) to leverage large language models for intelligent content transformation, making your chunks more useful for downstream AI applications.
108+
109+
### Document writer and storage
110+
111+
<xref:Microsoft.Extensions.DataIngestion.IngestionChunkWriter`1> stores processed chunks into a data store for later retrieval. Using Microsoft.Extensions.AI and [Microsoft.Extensions.VectorData.Abstractions](https://www.nuget.org/packages/Microsoft.Extensions.VectorData.Abstractions), the library provides the <xref:Microsoft.Extensions.DataIngestion.VectorStoreWriter`1> class that supports storing chunks in any vector store supported by Microsoft.Extensions.VectorData.
112+
113+
Vectore stores include popular options like [Qdrant](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.Qdrant), [SQL Server](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.SqlServer), [CosmosDB](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.CosmosNoSQL), [MongoDB](https://www.nuget.org/packages/Microsoft.SemanticKernel.Connectors.MongoDB), [ElasticSearch](https://www.nuget.org/packages/Elastic.SemanticKernel.Connectors.Elasticsearch), and many more. The writer can also automatically generate embeddings for your chunks using Microsoft.Extensions.AI, readying them for semantic search and retrieval scenarios.
114+
115+
```csharp
116+
OpenAIClient openAIClient = new(
117+
new ApiKeyCredential(Environment.GetEnvironmentVariable("GITHUB_TOKEN")!),
118+
new OpenAIClientOptions { Endpoint = new Uri("https://models.github.ai/inference") });
119+
120+
IEmbeddingGenerator<string, Embedding<float>> embeddingGenerator =
121+
openAIClient.GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator();
122+
123+
using SqliteVectorStore vectorStore = new(
124+
"Data Source=vectors.db;Pooling=false",
125+
new()
126+
{
127+
EmbeddingGenerator = embeddingGenerator
128+
});
129+
130+
// The writer requires the embedding dimension count to be specified.
131+
// For OpenAI's `text-embedding-3-small`, the dimension count is 1536.
132+
using VectorStoreWriter<string> writer = new(vectorStore, dimensionCount: 1536);
133+
```
134+
135+
### Document processing pipeline
136+
137+
The <xref:Microsoft.Extensions.DataIngestion.IngestionPipeline`1> API allows you to chain together the various data ingestion components into a complete workflow. You can combine:
138+
139+
- **Readers** to load documents from various sources.
140+
- **Processors** to transform and enrich document content.
141+
- **Chunkers** to break documents into manageable pieces.
142+
- **Writers** to store the final results in your chosen data store.
143+
144+
This pipeline approach reduces boilerplate code and makes it easy to build, test, and maintain complex data ingestion workflows.
145+
146+
```csharp
147+
using IngestionPipeline<string> pipeline = new(reader, chunker, writer, loggerFactory: loggerFactory)
148+
{
149+
DocumentProcessors = { imageAlternativeTextEnricher },
150+
ChunkProcessors = { summaryEnricher }
151+
};
152+
153+
await foreach (var result in pipeline.ProcessAsync(new DirectoryInfo("."), searchPattern: "*.md"))
154+
{
155+
Console.WriteLine($"Completed processing '{result.DocumentId}'. Succeeded: '{result.Succeeded}'.");
156+
}
157+
```
158+
159+
A single document ingestion failure shouldn't fail the whole pipeline. That's why <xref:Microsoft.Extensions.DataIngestion.IngestionPipeline`1.ProcessAsync*?displayProperty=nameWithType> implements partial success by returning `IAsyncEnumerable<IngestionResult>`. The caller is responsible for handling any failures (for example, by retrying failed documents or stopping on first error).

‎docs/ai/media/data-ingestion/dataingestion.png‎

76.4 KB
Loading

‎docs/ai/toc.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@ items:
4444
href: conceptual/embeddings.md
4545
- name: Vector databases
4646
href: conceptual/vector-databases.md
47+
- name: Data ingestion
48+
href: conceptual/data-ingestion.md
4749
- name: Prompt engineering
4850
href: conceptual/prompt-engineering-dotnet.md
4951
- name: Chain-of-thought prompting

0 commit comments

Comments
 (0)