RAG LLN NLP and Layout model

laujan · laujan · commit 5db59ae0d9ed · 2023-12-15T13:37:07.000-08:00
diff --git a/articles/ai-services/document-intelligence/concept-retrieval-augumented-generation.md b/articles/ai-services/document-intelligence/concept-retrieval-augumented-generation.md
@@ -6,7 +6,7 @@ author: laujan
 manager: nitinme
 ms.service: azure-ai-document-intelligence
 ms.topic: conceptual
-ms.date: 12/11/2023
+ms.date: 12/15/2023
 ms.author: lajanuar
 monikerRange: '>=doc-intel-3.0.0'
 ---
@@ -15,31 +15,29 @@ monikerRange: '>=doc-intel-3.0.0'
 
 <!-- markdownlint-disable MD036 -->
 
-[!INCLUDE [applies to v4.0, v3.1 and v3.0](includes/applies-to-v-4-0-v3-1-v3-0.md)]
+**This content applies to:** ![checkmark](../media/yes-icon.png) **v4.0 (preview)** 
 
-Retrieval Augmented Generation (RAG) is an document generative AI solution that combines a pretrained Large Language Model (LLM) like ChatGPT with an external data retrieval system to generate an enhanced response incorporating new data outside of the original training data. Adding an information retrieval system enables you to chat with your documents, generate captivating content, and access the power of Azure OpenAI models for your data. You also have more control over the data used by the LLM as it formulates a response.
+Retrieval Augmented Generation (RAG) is an document generative AI solution that combines a pretrained Large Language Model (LLM) like ChatGPT with an external data retrieval system to generate an enhanced response incorporating new data outside of the original training data. Adding an information retrieval system to your applications enables you to chat with your documents, generate captivating content, and access the power of Azure OpenAI models for your data. You also have more control over the data used by the LLM as it formulates a response.
 
-Text data chunking strategies play a key role in optimizing the RAG response and performance. Fixed-sized and semantic are two distinct chunking methods:
-
-* **Fixed-sized chunking**. Most chunking strategies used in RAG today are based on fix-sized text segments known as chunks. It is quick, easy, and effective with text that does not have a strong sematic structure such as logs and data. However it is not recommended for text that requires semantic understanding and precise context. The fixed-size nature of the window can result in severing words, sentences, or paragraphs impeding comprehension and disrupt the flow of information and understanding.
+## Azure AI Document Intelligence Layout model
 
-* **Semantic chunking**. This method divides the text into chunks based on semantic understanding that focuses on the subject or topic of a sentence. This method uses significant computational resources and is algorithmically complex. However, it has the distinct advantage of maintaining semantic consistency within each chunk.  It is particularly useful for text summarization, sentiment analysis, and document classification tasks. For example, if you are looking for a specific section in a document, you can use semantic chunking to divide the document into smaller chunks based on the section headers. This can help you to find the section you are looking for quickly and easily. An effective semantic chunking strategy yields the following benefits:
+The ADocument Intelligence [Layout model](concept-layout.md) is an advanced machine-learning based document analysis API. Using semantic chunking, the layout model offers a comprehensive solution for advanced content extraction and document structure analysis capabilities. With the layout model, you can easily extract paragraphs, tables, titles, section headings, selection marks, font/style properties, key-value pairs, math formulas, QR code/barcode and more from various document types. This enables you to divide a large body of texts or documents into smaller, meaningful chunks based on semantic content rather than arbitrary splits. The extracted information can be conveniently outputted to Markdown format, enabling you to define your semantic chunking strategy based on the provided building blocks.
 
-* **Efficient storage and retrieval**. Relevant semantic units of information are stored fetched rather than arbitrary snippets.
-* **Preserved semantic relevance**. Retrieve passages that are contextually, linguistically, and semantically relevant to the query.
-* **Enhanced interpretability** – A model is interpretable if it can take inputs and consistently produce the same outputs. The higher the interpretability, the easier it is for humans to understand why certain decisions and to trust the model's accuracy. Interpretability aids in transparency and traceability of generated responses.
+:::image type="content" source="media/rag/azure-rag-processing.png" alt-text="Screenshot depicting semantic chunking with RAG using Azure AI Document Intelligence":::
 
-## Semantic chunking for a targeted RAG response
+## Layout model and semantic chunking
 
 Long sentences are challenging for natural language processing (NLP) applications. Especially when they are comprised of multiple clauses, complex noun or verb phrases, relative clauses, and parenthetical groupings. Just like the human beholder, an NLP system also needs to successfully keep track of all the presented dependencies.The goal of semantic chunking is to find semantically coherent fragments of a sentence representation. These fragments can then be processed independently and recombined as semantic representations without loss of information, interpretation, or semantic relevance. The inherent meaning of the text is used as a guide for the chunking process.
 
-Markdown is a structured and formatted markup language and a popular input for enabling semantic chunking in RAG (Retrieval augmented generation). You can use the Markdown content from the [Layout model](concept-layout.md) to split documents based on paragraph boundaries, create specific chunks for tables, and fine-tune your chunking strategy to improve the quality of the generated responses.
+Text data chunking strategies play a key role in optimizing the RAG response and performance. Fixed-sized and semantic are two distinct chunking methods:
 
-## Semantic chunking with Azure AI Document Intelligence Layout model
+* **Fixed-sized chunking**. Most chunking strategies used in RAG today are based on fix-sized text segments known as chunks. It is quick, easy, and effective with text that does not have a strong sematic structure such as logs and data. However it is not recommended for text that requires semantic understanding and precise context. The fixed-size nature of the window can result in severing words, sentences, or paragraphs impeding comprehension and disrupt the flow of information and understanding.
 
-The Azure AI Document Intelligence [Layout model](concept-layout.md) is an advanced machine-learning based document analysis API. The model offers a comprehensive solution for semantic chunking by providing advanced content extraction and document structure analysis capabilities. With this model, you can easily extract paragraphs, tables, titles, section headings, selection marks, font/style properties, key-value pairs, math formulas, QR code/barcode and more from various document types. The extracted information can be conveniently outputted to Markdown format, enabling you to define your semantic chunking strategy based on the provided building blocks.
+* **Semantic chunking**. This method divides the text into chunks based on semantic understanding that focuses on the subject or topic of a sentence and uses significant computational resources and is algorithmically complex. However, it has the distinct advantage of maintaining semantic consistency within each chunk.  It is particularly useful for text summarization, sentiment analysis, and document classification tasks. For example, if you are looking for a specific section in a document, you can use semantic chunking to divide the document into smaller chunks based on the section headers. This can help you to find the section you are looking for quickly and easily. An effective semantic chunking strategy yields the following benefits:
 
-:::image type="content" source="media/rag/azure-rag-processing.png" alt-text="Screenshot depicting semantic chunking with RAG using Azure AI Document Intelligence":::
+## Semantic chunking with Document Intelligence layout model
+
+Markdown is a structured and formatted markup language and a popular input for enabling semantic chunking in RAG (Retrieval augmented generation). You can use the Markdown content from the [Layout model](concept-layout.md) to split documents based on paragraph boundaries, create specific chunks for tables, and fine-tune your chunking strategy to improve the quality of the generated responses.
 
 ### Benefits of using the layout model
 
@@ -57,7 +55,7 @@ The Azure AI Document Intelligence [Layout model](concept-layout.md) is an advan
 
 :::image type="content" source="media/rag/markdown-table-output.png" alt-text="Screenshot of table processed by layout model and outputted to Markdown.":::
 
-## Get started with Azure AI Document Intelligence
+## Get started with semantic chunking
 
 **Ready to begin?**
 
@@ -103,38 +101,15 @@ You can follow the [Document Intelligence studio quickstart](quickstarts/try-doc
 
 ## Build document chat with semantic chunking
 
-* [Azure OpenAI on your data](../openai/concepts/use-your-data) enables you to run supported chat **Azure AI Document Intelligence** layout model is underlying OCR engine for **Azure OpenAI on your documents** to ingest document data. It chunks long text based on table tables and paragraphs. You can also customize your chunking strategy based with these [scripts](https://github.com/microsoft/sample-app-aoai-chatGPT/tree/main/scripts).
-
-* Azure AI Document Intelligence is now integrated with Langchain as one of its document loaders. You can use it to easily load the data, output to Markdown format, and then use This [cookbook]() shows a simple demo for RAG pattern with Azure AI Document Intelligence as document loader and Azure Search as retriever in Langchain.
-
-```
-
-from langchain.document_loaders.doc_intelligence import DocumentIntelligenceLoader
-
-from langchain.text_splitter import MarkdownHeaderTextSplitter
-
-# Initiate Azure AI Document Intelligence to load the document and split it into chunks
-
-loader = DocumentIntelligenceLoader(file_path=<your file path>, api_key = doc_intelligence_key, api_endpoint = doc_intelligence_endpoint)
-docs = loader.load()
-
-# text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
-headers_to_split_on = [
-    ("#", "Header 1"),
-    ("##", "Header 2"),
-    ("###", "Header 3"),
-]
-text_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
+* [Azure OpenAI on your data](../openai/concepts/use-your-data) enables you to run supported chat on your documents.  Azure OpenAI on your data leverages the Document Intelligence layout model to extract and parse document data by chunking long text based on table tables and paragraphs. You can also customize your chunking strategy using [Azure OpenAI sample scripts](https://github.com/microsoft/sample-app-aoai-chatGPT/tree/main/scripts) located in our GitHub repo.
 
-splits = text_splitter.split_text(docs_string)
-splits
-```
+* Azure AI Document Intelligence is now integrated with [LangChain](https://python.langchain.com/docs/integrations/document_loaders/azure_document_intelligence) as one of its document loaders. You can use it to easily load the data, output to Markdown format, and then use This [notebook](https://microsoft.github.io/SynapseML/docs/Explore%20Algorithms/AI%20Services/Quickstart%20-%20Document%20Question%20and%20Answering%20with%20PDFs/) shows a simple demo for RAG pattern with Azure AI Document Intelligence as document loader and Azure Search as retriever in LangChain.
 
-* This [solution accelerator](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator) demonstrates an end-to-end baseline RAG pattern sample that uses Azure AI Search as a retriever and Azure AI Document Intelligence for document loading and semantic chunking.
+* The [chat with your data solution accelerator](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator) demonstrates an end-to-end baseline RAG pattern sample that uses Azure AI Search as a retriever and Azure AI Document Intelligence for document loading and semantic chunking.
 
-# Next steps
+## Next steps
 
-- Learn more about [Azure AI Document Intelligence](overview.md).
+* Learn more about [Azure AI Document Intelligence](overview.md).
 
 * [Learn how to process your own forms and documents](quickstarts/try-document-intelligence-studio.md) with the [Document Intelligence Studio](https://formrecognizer.appliedai.azure.com/studio).
 
