Skip to content

Commit 2af47bb

Browse files
TheovanKraayTheovanKraay
committed
semantic search demo
1 parent 99af0f0 commit 2af47bb

File tree

9 files changed

+1424
-0
lines changed

9 files changed

+1424
-0
lines changed

‎Python/CosmosDB-NoSQL_SemanticSearchDemo/.env.template‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Azure Cosmos DB Configuration (Keyless Authentication)
2+
COSMOS_URI=your_cosmos_db_uri_here
3+
COSMOS_DB_DATABASE=your_database_name
4+
5+
# Azure OpenAI Configuration
6+
AZURE_OPENAI_API_KEY=your_openai_api_key_here
7+
AZURE_OPENAI_ENDPOINT=https://your-openai-resource.openai.azure.com/
8+
9+
# Azure Cosmos DB Semantic Reranker (Private Preview)
10+
AZURE_COSMOS_SEMANTIC_RERANKER_INFERENCE_ENDPOINT=https://your-reranker-endpoint.dbinference.azure.com
11+
12+
# Note: The following variables are no longer needed with keyless authentication:
13+
# - COSMOS_FABCON_KEY (replaced with DefaultAzureCredential)
14+
# - AZURE_TENANT_ID (not needed for DefaultAzureCredential)
15+
# - AZURE_CLIENT_ID (not needed for DefaultAzureCredential)
16+
# - RERANKER_ENDPOINT (replaced with built-in Cosmos SDK semantic reranking)
17+
# - MSI_ENDPOINT (automatically set in Azure environments)
18+
# - IDENTITY_ENDPOINT (automatically set in Azure environments)

‎Python/CosmosDB-NoSQL_SemanticSearchDemo/.gitignore‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
.env
2+
.diskann
3+
.DS_Store
4+
.azure
5+
.idea
6+
.deployment
7+
.vscode
8+
venv
9+
__pycache__/
10+
*.pyc
11+
*.pyo

‎Python/CosmosDB-NoSQL_SemanticSearchDemo/README.md‎

Lines changed: 365 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,365 @@
1+
# Semantic Search Demo for Azure Cosmos DB
2+
3+
This repository contains a Python Streamlit application that demonstrates advanced search capabilities using Azure Cosmos DB, OpenAI embeddings, and semantic reranking. The application showcases vector search, full text search, text ranking, and hybrid search with intelligent semantic reranking to improve result relevance.
4+
5+
![screenshot](media/screen-shot.png)
6+
7+
## ✨ Features
8+
9+
- **Multi-Modal Search Integration** with Azure Cosmos DB:
10+
- 🔍 **Semantic search** for movies using OpenAI embeddings
11+
- 📝 **Full text search** with advanced text processing
12+
- 🔄 **Hybrid search** combining semantic and full text search
13+
- 📊 **Text ranking** for enhanced result ordering
14+
- **🎯 Semantic Reranking** (NEW):
15+
- Built-in Azure Cosmos DB SDK semantic reranking
16+
- Interactive UI toggle to enable/disable reranking
17+
- Preserves original metadata while improving result relevance
18+
- Uses DefaultAzureCredential for secure authentication
19+
- **📈 Multiple Index Support**:
20+
- No Index baseline
21+
- QFLAT vector index for balanced performance
22+
- DiskANN vector index for high-scale scenarios
23+
- **🛡️ Robust Error Handling**:
24+
- Graceful credential validation
25+
- Helpful troubleshooting messages
26+
- Fallback modes when services are unavailable
27+
- **🎨 Interactive UI** built with Streamlit
28+
29+
## Prerequisites
30+
31+
- [Azure Cosmos DB](https://azure.microsoft.com/services/cosmos-db/) account with NoSQL API with vector search and full text search enabled
32+
- [Azure OpenAI](https://azure.microsoft.com/products/ai-services/openai-service) account
33+
- [Azure CLI](https://docs.microsoft.com/cli/azure/install-azure-cli) (for local authentication)
34+
35+
## 🚀 Quick Start
36+
37+
### 1. Clone and Setup
38+
39+
```sh
40+
git clone https://github.com/TheovanKraay/AzureDataRetrievalAugmentedGenerationSamples.git
41+
cd AzureDataRetrievalAugmentedGenerationSamples/Python/CosmosDB-NoSQL_SemanticSearchDemo
42+
```
43+
44+
### 2. Create and Activate Virtual Environment
45+
46+
Create a virtual environment at the project root:
47+
48+
```sh
49+
python -m venv venv
50+
```
51+
52+
**Activate the virtual environment:**
53+
54+
**PowerShell (Windows):**
55+
```powershell
56+
.\venv\Scripts\Activate.ps1
57+
```
58+
59+
**Bash (Linux/macOS):**
60+
```bash
61+
source venv/bin/activate
62+
```
63+
64+
### 3. Install Dependencies
65+
66+
```sh
67+
pip install -r src/app/requirements.txt
68+
```
69+
70+
### 4. Configure Environment Variables
71+
72+
Copy the template and configure your settings:
73+
74+
```sh
75+
cp .env.template .env
76+
```
77+
78+
Edit `.env` with your actual values:
79+
80+
```properties
81+
# Azure Cosmos DB Configuration (Keyless Authentication)
82+
COSMOS_URI=https://your-account.documents.azure.com:443/
83+
COSMOS_DB_DATABASE=your_database_name
84+
85+
# Azure OpenAI Configuration
86+
AZURE_OPENAI_API_KEY=your_openai_api_key_here
87+
AZURE_OPENAI_ENDPOINT=https://your-openai-resource.openai.azure.com/
88+
89+
# Azure Cosmos DB Semantic Reranker (Private Preview)
90+
AZURE_COSMOS_SEMANTIC_RERANKER_INFERENCE_ENDPOINT=https://your-reranker-endpoint.dbinference.azure.com
91+
```
92+
93+
### 5. Authentication Setup
94+
95+
The app uses **keyless authentication** with Azure Cosmos DB:
96+
97+
**🆔 DefaultAzureCredential (Recommended)**:
98+
- Uses `DefaultAzureCredential` for secure authentication
99+
- **Local Development**: Uses Azure CLI (`az login`)
100+
- **Azure Deployment**: Uses Managed Identity automatically
101+
- **Required**: Your identity needs "Cosmos DB Built-in Data Contributor" role
102+
- **Benefits**: More secure, no secrets in configuration
103+
104+
### 6. Authenticate with Azure (Local Development)
105+
106+
```sh
107+
az login
108+
```
109+
110+
### 7. Run the Application
111+
112+
Make sure your virtual environment is activated, then choose one of the options below:
113+
114+
#### Option A: Without Semantic Reranker (Standard Features)
115+
116+
**PowerShell (Windows):**
117+
```powershell
118+
streamlit run src/app/cosmos-app.py --server.port 8501
119+
```
120+
121+
**Bash (Linux/macOS):**
122+
```bash
123+
streamlit run src/app/cosmos-app.py --server.port 8501
124+
```
125+
126+
#### Option B: With Semantic Reranker (Private Preview)
127+
128+
> **⚠️ PRIVATE PREVIEW**: The Azure Cosmos DB Semantic Reranker is currently in private preview. To enable this feature, please reach out to the Azure Cosmos DB team for access and to obtain your reranker endpoint.
129+
130+
**PowerShell (Windows):**
131+
```powershell
132+
$env:AZURE_COSMOS_SEMANTIC_RERANKER_INFERENCE_ENDPOINT="https://your-reranker-endpoint.dbinference.azure.com"; streamlit run src/app/cosmos-app.py --server.port 8501
133+
```
134+
135+
**Bash (Linux/macOS):**
136+
```bash
137+
export AZURE_COSMOS_SEMANTIC_RERANKER_INFERENCE_ENDPOINT="https://your-reranker-endpoint.dbinference.azure.com"
138+
streamlit run src/app/cosmos-app.py --server.port 8501
139+
```
140+
141+
> **💡 Note**: Replace `https://your-reranker-endpoint.dbinference.azure.com` with your actual semantic reranker endpoint URL provided by the Azure Cosmos DB team.
142+
143+
## 🎯 Using the Semantic Reranker (Private Preview)
144+
145+
> **⚠️ PRIVATE PREVIEW**: The Azure Cosmos DB Semantic Reranker is currently in private preview. Contact the Azure Cosmos DB team to request access and obtain your reranker endpoint.
146+
147+
The application includes Azure Cosmos DB's built-in semantic reranker for improved search relevance:
148+
149+
### Features:
150+
- **Built-in Integration**: Uses Azure Cosmos DB SDK `semantic_rerank()` method
151+
- **Smart Ranking**: Reorders search results based on semantic similarity to your query
152+
- **UI Toggle**: Enable/disable reranking with the checkbox in the sidebar
153+
- **Metadata Preservation**: Maintains all original result data (IDs, titles, scores)
154+
- **Seamless Authentication**: Uses the same DefaultAzureCredential as Cosmos DB
155+
156+
### How to Use:
157+
1. **Ensure you have access**: Contact the Azure Cosmos DB team for private preview access
158+
2. **Get your endpoint**: Obtain your semantic reranker inference endpoint URL
159+
3. **Start the app**: Use Option B from the "Run the Application" section above
160+
4. **Check the "Use Semantic Reranker" checkbox** in the sidebar
161+
162+
### How to Use:
163+
1. **Start the application** following the instructions above
164+
2. **Check the "Use Semantic Reranker" checkbox** in the sidebar
165+
3. **Perform any search** (vector, text, or hybrid)
166+
4. **Compare results** with and without reranking enabled
167+
5. **View reranking status** in the results display
168+
169+
### Requirements:
170+
- **Authentication**: Uses your existing Cosmos DB credentials (DefaultAzureCredential)
171+
- **Azure Cosmos DB**: Account must support semantic reranking functionality
172+
173+
### Benefits:
174+
- **Built-in SDK Integration**: No external API calls required
175+
- **Consistent Authentication**: Uses your existing Cosmos DB credentials
176+
- **Automatic Fallback**: App continues working even if reranking fails
177+
178+
## 📁 Project Structure
179+
180+
```
181+
CosmosDB-NoSQL_SemanticSearchDemo/
182+
├── src/
183+
│ ├── app/
184+
│ │ ├── cosmos-app.py # Main Streamlit application
185+
│ │ ├── requirements.txt # Python dependencies
186+
│ │ ├── .env.template # Environment variable template
187+
│ │ ├── .env # Your credentials (not in git)
188+
│ │ └── venv/ # Local virtual environment (optional)
189+
│ └── data/
190+
│ ├── data-loader.py # Data ingestion script
191+
│ └── drop-containers.py # Container cleanup utility
192+
├── venv/ # Project virtual environment
193+
├── README.md
194+
└── .gitignore
195+
```
196+
197+
## 🔧 Advanced Configuration
198+
199+
### Environment Variables Reference:
200+
201+
| Variable | Description | Required |
202+
|----------|-------------|----------|
203+
| `COSMOS_URI` | Azure Cosmos DB endpoint | ✅ |
204+
| `COSMOS_DB_DATABASE` | Azure Cosmos DB database name | ✅ |
205+
| `AZURE_OPENAI_API_KEY` | Azure OpenAI API key | ✅ |
206+
| `OPENAI_ENDPOINT` | Azure OpenAI endpoint | ✅ |
207+
208+
*✅ = Required*
209+
210+
**Note**: Semantic reranking uses the built-in Cosmos DB SDK with your existing authentication credentials.
211+
212+
### Authentication:
213+
214+
**Cosmos DB & Semantic Reranker Authentication:**
215+
- **Local Development**: Uses Azure CLI authentication (`az login`)
216+
- **Azure Deployment**: Uses Managed Identity authentication automatically
217+
- **DefaultAzureCredential**: Handles authentication flow seamlessly
218+
- **Required Role**: "Cosmos DB Built-in Data Contributor" on your Cosmos DB account
219+
220+
## 🚀 Deploy to Azure with VS Code
221+
222+
### Prerequisites:
223+
- Azure subscription with App Service capabilities
224+
- VS Code with Azure App Service extension
225+
226+
### Steps:
227+
228+
1. **Install the Azure App Service extension**:
229+
- Open Extensions view (Ctrl+Shift+X)
230+
- Search for "Azure App Service" and install
231+
232+
2. **Create Azure Web App**:
233+
- Create [Azure Web App](https://learn.microsoft.com/azure/app-service/overview)
234+
- Use Linux service plan (B1 SKU or higher)
235+
- Select Python 3.10+ runtime
236+
237+
3. **Configure App Service**:
238+
- Go to Configuration → General Settings
239+
- Set **Startup Command**:
240+
```shell
241+
python -m pip install -r src/app/requirements.txt && python -m streamlit run src/app/cosmos-app.py --server.port 8000 --server.address 0.0.0.0
242+
```
243+
- Add environment variables in Configuration → Application Settings
244+
245+
4. **Deploy**:
246+
- Press Ctrl+Shift+P
247+
- Select "Azure App Service: Deploy to Web App"
248+
- Select this project folder
249+
- Choose your subscription and Web App
250+
- Wait for deployment (up to 5 minutes)
251+
252+
### 🔐 Production Security:
253+
- Use **Managed Identity** for Azure service authentication
254+
- Store sensitive values in **Azure Key Vault**
255+
- Configure **Application Settings** instead of .env files
256+
257+
## 📊 Loading Data into Cosmos DB
258+
259+
The app automatically creates containers with proper vector and text search policies. Use the data loader to populate with your data:
260+
261+
### Prerequisites for Data Loading:
262+
1. **Environment Variables**: Ensure your `.env` file is configured with:
263+
- `COSMOS_URI`: Your Cosmos DB endpoint
264+
- `COSMOS_DB_DATABASE`: Target database name
265+
- `AZURE_OPENAI_API_KEY`: Your Azure OpenAI API key
266+
- `AZURE_OPENAI_ENDPOINT`: Your Azure OpenAI endpoint
267+
268+
2. **Virtual Environment**: Make sure your virtual environment is activated:
269+
```sh
270+
.\venv\Scripts\Activate.ps1 # Windows PowerShell
271+
# or
272+
source venv/bin/activate # Linux/macOS
273+
```
274+
275+
### Data Loader Arguments:
276+
- `--text_field_name`: Field containing text for embedding generation (required)
277+
- `--path_to_json_array`: Path to JSON file or URL containing data array (required)
278+
- `--database_name`: Cosmos DB database name (required)
279+
- `--concurrency`: Number of concurrent operations (default: 1, recommended: 10-20)
280+
- `--vector_field_name`: Field with pre-generated embeddings (optional)
281+
- `--re_embed`: Whether to regenerate embeddings (default: False)
282+
283+
### Basic Usage:
284+
```sh
285+
cd src/data
286+
python data-loader.py \
287+
--text_field_name "overview" \
288+
--path_to_json_array "your-data.json" \
289+
--database_name "searchdemo" \
290+
--concurrency 20 \
291+
--re_embed True
292+
```
293+
294+
### Movie Dataset Example:
295+
```sh
296+
python src/data/data-loader.py \
297+
--text_field_name "overview" \
298+
--path_to_json_array "https://raw.githubusercontent.com/microsoft/AzureDataRetrievalAugmentedGenerationSamples/refs/heads/main/DataSet/Movies/MovieLens-4489-256D.json" \
299+
--database_name "searchdemo" \
300+
--concurrency 20 \
301+
--vector_field_name "vector" \
302+
--re_embed True
303+
```
304+
305+
### Using Pre-existing Embeddings:
306+
If your data already contains vector embeddings:
307+
```sh
308+
python src/data/data-loader.py \
309+
--text_field_name "text" \
310+
--path_to_json_array "data-with-vectors.json" \
311+
--database_name "searchdemo" \
312+
--vector_field_name "embedding" \
313+
--concurrency 10 \
314+
--re_embed False
315+
```
316+
317+
> **💡 Note**: Use the same database name that you configured in the `COSMOS_DB_DATABASE` environment variable.
318+
319+
### 🗑️ Container Management:
320+
Use the cleanup utility to remove containers:
321+
```sh
322+
python src/data/drop-containers.py
323+
```
324+
325+
## 🤝 Contributing
326+
327+
1. Fork the repository
328+
2. Create a feature branch: `git checkout -b feature/amazing-feature`
329+
3. Commit changes: `git commit -m 'Add amazing feature'`
330+
4. Push to branch: `git push origin feature/amazing-feature`
331+
5. Open a Pull Request
332+
333+
## 📝 License
334+
335+
This project is licensed under the MIT License - see the LICENSE file for details.
336+
337+
## 🆘 Support & Troubleshooting
338+
339+
### Common Issues:
340+
341+
**❌ "Failed to connect to Cosmos DB"**
342+
- Verify `COSMOS_URI` in .env
343+
- Run `az login` to authenticate
344+
- Check that your account has "Cosmos DB Built-in Data Contributor" role
345+
- Check network connectivity to Azure
346+
347+
**❌ "Semantic reranking failed"**
348+
- Ensure you're authenticated: `az login`
349+
- Verify your account has access to the Cosmos DB account
350+
- Check that the Cosmos DB account supports semantic reranking
351+
- App will continue working with original search results
352+
353+
**❌ "No results found"**
354+
- Ensure data is loaded into Cosmos containers
355+
- Check container names match the app configuration
356+
357+
### Performance Tips:
358+
- Use **DiskANN index** for large datasets (>100K documents)
359+
- Use **QFLAT index** for balanced performance and cost
360+
- Adjust **concurrency** in data loader based on Cosmos DB throughput
361+
362+
### 📞 Need Help?
363+
- Check the [Issues](https://github.com/TheovanKraay/AzureDataRetrievalAugmentedGenerationSamples/issues) page
364+
- Review Azure Cosmos DB [documentation](https://docs.microsoft.com/azure/cosmos-db/)
365+
- Consult Azure OpenAI [best practices](https://docs.microsoft.com/azure/cognitive-services/openai/)

‎Python/CosmosDB-NoSQL_SemanticSearchDemo/media/screen-shot.png‎

455 KB
Loading

0 commit comments

Comments
 (0)