docs: revise the docs for Azure Blob source

Author: georgeh0

docs/docs/ops/sources.md

@@ -148,6 +148,72 @@ The spec takes the following fields:
 ### Schema
 
 The output is a [*KTable*](/docs/core/data_types#ktable) with the following sub fields:
+
+*   `filename` (*Str*, key): the filename of the file, including the path, relative to the root directory, e.g. `"dir1/file1.md"`.
+*   `content` (*Str* if `binary` is `False`, otherwise *Bytes*): the content of the file.
+
+
+## AzureBlob
+
+The `AzureBlob` source imports files from Azure Blob Storage.
+
+### Setup for Azure Blob Storage
+
+#### Get Started
+
+If you didn't have experience with Azure Blob Storage, you can refer to the [quickstart](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal).
+These are actions you need to take:
+
+*   Create a storage account in the [Azure Portal](https://portal.azure.com/).
+*   Create a container in the storage account.
+*   Upload your files to the container.
+*   Grant the user / identity / service principal (depends on your authentication method, see below) access to the container. At minimum, a **Storage Blob Data Reader** role is needed. See [this doc](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-data-operations-portal) for reference.
+
+#### Authentication
+
+We use Azure’s **Default Credential** system (DefaultAzureCredential) for secure and flexible authentication.
+This allows you to connect to Azure services without putting any secrets in the code or flow spec.
+It automatically chooses the best authentication method based on your environment:
+
+*   On your local machine: uses your Azure CLI login (`az login`) or environment variables.
+
+    ```sh
+    az login
+    # Optional: Set a default subscription if you have more than one
+    az account set --subscription "<YOUR_SUBSCRIPTION_NAME_OR_ID>"
+    ```
+*   In Azure (VM, App Service, AKS, etc.): uses the resource’s Managed Identity.
+*   In automated environments: supports Service Principals via environment variables
+    *   `AZURE_CLIENT_ID`
+    *   `AZURE_TENANT_ID`
+    *   `AZURE_CLIENT_SECRET`
+
+You can refer to [this doc](https://learn.microsoft.com/en-us/azure/developer/python/sdk/authentication/overview) for more details.
+
+### Spec
+
+The spec takes the following fields:
+
+*   `account_name` (`str`): the name of the storage account.
+*   `container_name` (`str`): the name of the container.
+*   `prefix` (`str`, optional): if provided, only files with path starting with this prefix will be imported.
+*   `binary` (`bool`, optional): whether reading files as binary (instead of text).
+*   `included_patterns` (`list[str]`, optional): a list of glob patterns to include files, e.g. `["*.txt", "docs/**/*.md"]`.
+    If not specified, all files will be included.
+*   `excluded_patterns` (`list[str]`, optional): a list of glob patterns to exclude files, e.g. `["*.tmp", "**/*.log"]`.
+    Any file or directory matching these patterns will be excluded even if they match `included_patterns`.
+    If not specified, no files will be excluded.
+
+    :::info
+
+    `included_patterns` and `excluded_patterns` are using Unix-style glob syntax. See [globset syntax](https://docs.rs/globset/latest/globset/index.html#syntax) for the details.
+
+    :::
+
+### Schema
+
+The output is a [*KTable*](/docs/core/data_types#ktable) with the following sub fields:
+
 *   `filename` (*Str*, key): the filename of the file, including the path, relative to the root directory, e.g. `"dir1/file1.md"`.
 *   `content` (*Str* if `binary` is `False`, otherwise *Bytes*): the content of the file.
 

examples/azure_blob_embedding/.env.example

@@ -5,18 +5,3 @@ COCOINDEX_DATABASE_URL=postgres://cocoindex:cocoindex@localhost/cocoindex
 AZURE_STORAGE_ACCOUNT_NAME=testnamecocoindex1
 AZURE_BLOB_CONTAINER_NAME=testpublic1
 AZURE_BLOB_PREFIX=
-
-# Authentication Options (choose ONE - in priority order):
-
-# Option 1: Connection String (HIGHEST PRIORITY - recommended for development)
-# NOTE: Use ACCOUNT KEY connection string, NOT SAS connection string!
-# AZURE_BLOB_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=testnamecocoindex1;AccountKey=key1-goes-here;EndpointSuffix=core.windows.net
-
-# Option 2: SAS Token (SECOND PRIORITY - recommended for production)
-# AZURE_BLOB_SAS_TOKEN=sp=r&st=2024-01-01T00:00:00Z&se=2025-12-31T23:59:59Z&spr=https&sv=2022-11-02&sr=c&sig=...
-
-# Option 3: Account Key (THIRD PRIORITY)
-# AZURE_BLOB_ACCOUNT_KEY=key1-goes-here
-
-# Option 4: Anonymous access (FALLBACK - for public containers only)
-# Leave all auth options commented out - testpublic1 container supports this!

examples/azure_blob_embedding/README.md

@@ -2,48 +2,31 @@ This example builds an embedding index based on files stored in an Azure Blob St
 It continuously updates the index as files are added / updated / deleted in the source container:
 it keeps the index in sync with the Azure Blob Storage container effortlessly.
 
-## Quick Start (Public Test Container)
-
-🚀 **Try it immediately!** We provide a public test container with sample documents:
-- **Account:** `testnamecocoindex1`
-- **Container:** `testpublic1` (public access)
-- **No authentication required!**
-
-Just copy `.env.example` to `.env` and run - it works out of the box with anonymous access.
-
 ## Prerequisite
 
 Before running the example, you need to:
 
 1.  [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one.
 
 2.  Prepare for Azure Blob Storage.
-    You'll need an Azure Storage account and container. Supported authentication methods:
-    - **Connection String** (recommended for development)
-    - **SAS Token** (recommended for production)
-    - **Account Key** (full access)
-    - **Anonymous access** (for public containers only)
+    See [Setup for Azure Blob Storage](https://cocoindex.io/docs/ops/sources#setup-for-azure-blob-storage) for more details.
 
-3.  Create a `.env` file with your Azure Blob Storage configuration.
-    Start from copying the `.env.example`, and then edit it to fill in your credentials.
+3.  Create a `.env` file with your Azure Blob Storage container name and (optionally) prefix.
+    Start from copying the `.env.example`, and then edit it to fill in your bucket name and prefix.
 
     ```bash
     cp .env.example .env
     $EDITOR .env
     ```
 
-    Example `.env` file with connection string:
+    Example `.env` file:
     ```
     # Database Configuration
     DATABASE_URL=postgresql://localhost:5432/cocoindex
 
     # Azure Blob Storage Configuration
-    AZURE_STORAGE_ACCOUNT_NAME=mystorageaccount
-    AZURE_BLOB_CONTAINER_NAME=mydocuments
-    AZURE_BLOB_PREFIX=
-
-    # Authentication (choose one)
-    AZURE_BLOB_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=mykey123;EndpointSuffix=core.windows.net
+    AZURE_BLOB_STORAGE_ACCOUNT_NAME=your-account-name
+    AZURE_BLOB_STORAGE_CONTAINER_NAME=your-container-name
     ```
 
 ## Run
@@ -60,7 +43,7 @@ Run:
 python main.py
 ```
 
-During running, it will keep observing changes in the Azure Blob Storage container and update the index automatically.
+During running, it will keep observing changes in the Amazon S3 bucket and update the index automatically.
 At the same time, it accepts queries from the terminal, and performs search on top of the up-to-date index.
 
 
@@ -80,74 +63,3 @@ cocoindex server -ci -L main.py
 ```
 
 Then open the CocoInsight UI at [https://cocoindex.io/cocoinsight](https://cocoindex.io/cocoinsight).
-
-## Authentication Methods & Troubleshooting
-
-### Connection String (Recommended for Development)
-```bash
-AZURE_BLOB_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=testnamecocoindex1;AccountKey=your-key;EndpointSuffix=core.windows.net"
-```
-- **Pros:** Easiest to set up, contains all necessary information
-- **Cons:** Contains account key (full access)
-- **⚠️ Important:** Use **Account Key** connection string, NOT SAS connection string!
-
-### SAS Token (Recommended for Production)
-```bash
-AZURE_BLOB_SAS_TOKEN="sp=r&st=2024-01-01T00:00:00Z&se=2025-12-31T23:59:59Z&spr=https&sv=2022-11-02&sr=c&sig=..."
-```
-- **Pros:** Fine-grained permissions, time-limited
-- **Cons:** More complex to generate and manage
-
-**SAS Token Requirements:**
-- `sp=r` - Read permission (required)
-- `sp=rl` - Read + List permissions (recommended)
-- `sr=c` - Container scope (to access all blobs)
-- Valid time range (`st` and `se` in UTC)
-
-### Account Key
-```bash
-AZURE_BLOB_ACCOUNT_KEY="your-account-key-here"
-```
-- **Pros:** Simple to use
-- **Cons:** Full account access, security risk
-
-### Anonymous Access
-Leave all authentication options empty - only works with public containers.
-
-## Common Issues
-
-### 401 Authentication Error
-```
-Error: server returned error status which will not be retried: 401
-Error Code: NoAuthenticationInformation
-```
-
-**Solutions:**
-1. **Check authentication priority:** Connection String > SAS Token > Account Key > Anonymous
-2. **Verify SAS token permissions:** Must include `r` (read) and `l` (list) permissions
-3. **Check SAS token expiry:** Ensure `se` (expiry time) is in the future
-4. **Verify container scope:** Use `sr=c` for container-level access
-
-### Connection String Issues
-
-**⚠️ CRITICAL: Use Account Key Connection String, NOT SAS Connection String!**
-
-**✅ Correct (Account Key Connection String):**
-```
-DefaultEndpointsProtocol=https;AccountName=testnamecocoindex1;AccountKey=your-key;EndpointSuffix=core.windows.net
-```
-
-**❌ Wrong (SAS Connection String - will not work):**
-```
-BlobEndpoint=https://testnamecocoindex1.blob.core.windows.net/;SharedAccessSignature=sp=r&st=...
-```
-
-**Other tips:**
-- Don't include quotes in the actual connection string value
-- Account name in connection string should match `AZURE_STORAGE_ACCOUNT_NAME`
-- Connection string must contain `AccountKey=` parameter
-
-### Container Access Issues
-- Verify container exists and account has access
-- Check `AZURE_BLOB_CONTAINER_NAME` spelling
-- For anonymous access, container must be public

examples/azure_blob_embedding/main.py

@@ -101,20 +101,22 @@ def _main() -> None:
     pool = ConnectionPool(os.getenv("COCOINDEX_DATABASE_URL"))
 
     azure_blob_text_embedding_flow.setup()
-    with cocoindex.FlowLiveUpdater(azure_blob_text_embedding_flow):
-        # Run queries in a loop to demonstrate the query capabilities.
-        while True:
-            query = input("Enter search query (or Enter to quit): ")
-            if query == "":
-                break
-            # Run the query function with the database connection pool and the query.
-            results = search(pool, query)
-            print("\nSearch results:")
-            for result in results:
-                print(f"[{result['score']:.3f}] {result['filename']}")
-                print(f"    {result['text']}")
-                print("---")
-            print()
+    update_stats = azure_blob_text_embedding_flow.update()
+    print(update_stats)
+
+    # Run queries in a loop to demonstrate the query capabilities.
+    while True:
+        query = input("Enter search query (or Enter to quit): ")
+        if query == "":
+            break
+        # Run the query function with the database connection pool and the query.
+        results = search(pool, query)
+        print("\nSearch results:")
+        for result in results:
+            print(f"[{result['score']:.3f}] {result['filename']}")
+            print(f"    {result['text']}")
+            print("---")
+        print()
 
 
 if __name__ == "__main__":