Remove duplicate samples

Author: pamelafox

README.md

@@ -1,4 +1,25 @@
-# ChatGPT-like app with your data using Azure OpenAI and Azure AI Search (Python)
+<!--
+---
+name: RAG chat app with your data (Python)
+description: Chat with your domain data using Azure OpenAI and Azure AI Search.
+languages:
+- python
+- typescript
+- bicep
+- azdeveloper
+products:
+- azure-openai
+- azure-cognitive-search
+- azure-app-service
+- azure
+page_type: sample
+urlFragment: azure-search-openai-demo
+---
+-->
+
+# RAG chat app with Azure OpenAI and Azure AI Search (Python)
+
+This solution creates a ChatGPT-like frontend experience over your own documents using RAG (Retrieval Augmented Generation). It uses Azure OpenAI Service to access GPT models, and Azure AI Search for data indexing and retrieval.
 
 This solution's backend is written in Python. There are also [**JavaScript**](https://aka.ms/azai/js/code), [**.NET**](https://aka.ms/azai/net/code), and [**Java**](https://aka.ms/azai/java/code) samples based on this one. Learn more about [developing AI apps using Azure AI Services](https://aka.ms/azai).
 

docs/data_ingestion.md

@@ -1,19 +1,34 @@
-# Indexing documents for the Chat App
+# RAG chat: Data ingestion
 
-This guide provides more details for using the `prepdocs` script to index documents for the Chat App.
+The [azure-search-openai-demo](/) project can set up a full RAG chat app on Azure AI Search and OpenAI so that you can chat on custom data, like internal enterprise data or domain-specific knowledge sets. For full instructions on setting up the project, consult the [main README](/README.md), and then return here for detailed instructions on the data ingestion component.
 
-- [Supported document formats](#supported-document-formats)
-- [Overview of the manual indexing process](#overview-of-the-manual-indexing-process)
+The chat app provides two ways to ingest data: manual indexing and integrated vectorization. This document explains the differences between the two approaches and provides an overview of the manual indexing process.
+
+- [Manual indexing process](#manual-indexing-process)
+  - [Supported document formats](#supported-document-formats)
   - [Chunking](#chunking)
   - [Categorizing data for enhanced search](#enhancing-search-functionality-with-data-categorization)
   - [Indexing additional documents](#indexing-additional-documents)
   - [Removing documents](#removing-documents)
-- [Overview of Integrated Vectorization](#overview-of-integrated-vectorization)
+- [Integrated Vectorization](#integrated-vectorization)
   - [Indexing of additional documents](#indexing-of-additional-documents)
   - [Removal of documents](#removal-of-documents)
   - [Scheduled indexing](#scheduled-indexing)
 
-## Supported document formats
+## Manual indexing process
+
+The [`prepdocs.py`](../app/backend/prepdocs.py) script is responsible for both uploading and indexing documents. The typical usage is to call it using `scripts/prepdocs.sh` (Mac/Linux) or `scripts/prepdocs.ps1` (Windows), as these scripts will set up a Python virtual environment and pass in the required parameters based on the current `azd` environment. You can pass additional arguments directly to the script, for example `scripts/prepdocs.ps1 --removeall`. Whenever `azd up` or `azd provision` is run, the script is called automatically.
+
+![Diagram of the indexing process](images/diagram_prepdocs.png)
+
+The script uses the following steps to index documents:
+
+1. If it doesn't yet exist, create a new index in Azure AI Search.
+2. Upload the PDFs to Azure Blob Storage.
+3. Split the PDFs into chunks of text.
+4. Upload the chunks to Azure AI Search. If using vectors (the default), also compute the embeddings and upload those alongside the text.
+
+### Supported document formats
 
 In order to ingest a document format, we need a tool that can turn it into text. By default, use Azure Document Intelligence (DI in the table below), but we also have local parsers for several formats. The local parsers are not as sophisticated as Azure Document Intelligence, but they can be used to decrease charges.
 
@@ -29,33 +44,20 @@ In order to ingest a document format, we need a tool that can turn it into text.
 
 The Blob indexer used by the Integrated Vectorization approach also supports a few [additional formats](https://learn.microsoft.com/azure/search/search-howto-indexing-azure-blob-storage#supported-document-formats).
 
-## Overview of the manual indexing process
-
-The [`prepdocs.py`](../app/backend/prepdocs.py) script is responsible for both uploading and indexing documents. The typical usage is to call it using `scripts/prepdocs.sh` (Mac/Linux) or `scripts/prepdocs.ps1` (Windows), as these scripts will set up a Python virtual environment and pass in the required parameters based on the current `azd` environment. You can pass additional arguments directly to the script, for example `scripts/prepdocs.ps1 --removeall`. Whenever `azd up` or `azd provision` is run, the script is called automatically.
+### Chunking
 
-![Diagram of the indexing process](images/diagram_prepdocs.png)
+We're often asked why we need to break up the PDFs into chunks when Azure AI Search supports searching large documents.
 
-The script uses the following steps to index documents:
+Chunking allows us to limit the amount of information we send to OpenAI due to token limits. By breaking up the content, it allows us to easily find potential chunks of text that we can inject into OpenAI. The method of chunking we use leverages a sliding window of text such that sentences that end one chunk will start the next. This allows us to reduce the chance of losing the context of the text.
 
-1. If it doesn't yet exist, create a new index in Azure AI Search.
-2. Upload the PDFs to Azure Blob Storage.
-3. Split the PDFs into chunks of text.
-4. Upload the chunks to Azure AI Search. If using vectors (the default), also compute the embeddings and upload those alongside the text.
+If needed, you can modify the chunking algorithm in `scripts/prepdocslib/textsplitter.py`.
 
 ### Enhancing search functionality with data categorization
 
 To enhance search functionality, categorize data during the ingestion process with the `--category` argument, for example `scripts/prepdocs.ps1 --category ExampleCategoryName`. This argument specifies the category to which the data belongs, enabling you to filter search results based on these categories.
 
 After running the script with the desired category, ensure these categories are added to the 'Include Category' dropdown list. This can be found in the developer settings of [`Chat.tsx`](../app/frontend/src/pages/chat/Chat.tsx) and [`Ask.tsx`](../app/frontend/src/pages/ask/Ask.tsx). The default option for this dropdown is "All". By including specific categories, you can refine your search results more effectively.
 
-### Chunking
-
-We're often asked why we need to break up the PDFs into chunks when Azure AI Search supports searching large documents.
-
-Chunking allows us to limit the amount of information we send to OpenAI due to token limits. By breaking up the content, it allows us to easily find potential chunks of text that we can inject into OpenAI. The method of chunking we use leverages a sliding window of text such that sentences that end one chunk will start the next. This allows us to reduce the chance of losing the context of the text.
-
-If needed, you can modify the chunking algorithm in `scripts/prepdocslib/textsplitter.py`.
-
 ### Indexing additional documents
 
 To upload more PDFs, put them in the data/ folder and run `./scripts/prepdocs.sh` or `./scripts/prepdocs.ps1`.
@@ -70,7 +72,7 @@ To remove all documents, use `scripts/prepdocs.sh --removeall` or `scripts/prepd
 
 You can also remove individual documents by using the `--remove` flag. Open either `scripts/prepdocs.sh` or `scripts/prepdocs.ps1` and replace `/data/*` with `/data/YOUR-DOCUMENT-FILENAME-GOES-HERE.pdf`. Then run `scripts/prepdocs.sh --remove` or `scripts/prepdocs.ps1 --remove`.
 
-## Overview of Integrated Vectorization
+## Integrated Vectorization
 
 Azure AI Search includes an [integrated vectorization feature](https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/announcing-the-public-preview-of-integrated-vectorization-in/ba-p/3960809#:~:text=Integrated%20vectorization%20is%20a%20new%20feature%20of%20Azure,pull-indexers%2C%20and%20vectorization%20of%20text%20queries%20through%20vectorizers), a cloud-based approach to data ingestion. Integrated vectorization takes care of document format cracking, data extraction, chunking, vectorization, and indexing, all with Azure technologies.
 

docs/deploy_private.md

@@ -1,5 +1,25 @@
-
-# Deploying with private access
+<!--
+---
+name: RAG chat with private endpoints
+description: Configure access to a chat app so that it's only accessible from private endpoints.
+languages:
+- python
+- typescript
+- bicep
+- azdeveloper
+products:
+- azure-openai
+- azure-cognitive-search
+- azure-app-service
+- azure
+page_type: sample
+urlFragment: azure-search-openai-demo-private-access
+---
+-->
+
+# RAG chat: Deploying with private access
+
+The [azure-search-openai-demo](/) project can set up a full RAG chat app on Azure AI Search and OpenAI so that you can chat on custom data, like internal enterprise data or domain-specific knowledge sets. For full instructions on setting up the project, consult the [main README](/README.md), and then return here for detailed instructions on configuring private endpoints.
 
 ⚠️ This feature is not yet compatible with Azure Container Apps, so you will need to [deploy to Azure App Service](./azure_app_service.md) instead.
 

docs/login_and_acl.md

@@ -1,4 +1,25 @@
-# Setting up optional login and document level access control
+<!--
+---
+name: RAG chat with document security
+description: This guide demonstrates how to add an optional login and document level access control system to a RAG chat app for your domain data. This system can be used to restrict access to indexed data to specific users.
+languages:
+- python
+- typescript
+- bicep
+- azdeveloper
+products:
+- azure-openai
+- azure-cognitive-search
+- azure-app-service
+- azure
+page_type: sample
+urlFragment: azure-search-openai-demo-document-security
+---
+-->
+
+# RAG chat: Setting up optional login and document level access control
+
+The [azure-search-openai-demo](/) project can set up a full RAG chat app on Azure AI Search and OpenAI so that you can chat on custom data, like internal enterprise data or domain-specific knowledge sets. For full instructions on setting up the project, consult the [main README](/README.md), and then return here for detailed instructions on configuring login and access control.
 
 ## Table of Contents