docs: Cloud Object Storage Update

docs/blog/0.64.md

@@ -28,7 +28,7 @@ You can now search directly within the row and column add dialogs. It was alread
 When you add a local filter to a widget in a Canvas dashboard, it will now be represented as an inline filter pill inside the widget. This helps end users understand which filters are active for that specific widget.
 
 ## Support for AWS STS Authentication in Connectors
-We now support AWS STS authentication for our DuckDB and S3 connectors. You can find more details in our [documentation](/build/connectors/data-source/s3#iam-role-based-authentication).
+We now support AWS STS authentication for our DuckDB and S3 connectors. You can find more details in our [documentation](/build/connectors/data-source/s3#method-2-iam-role-assumption).
 
 ## Cell Inspector for Long Dimension Values
 Sometimes, long dimension values are truncated in leaderboards and tables. You can now press the spacebar to toggle the Cell Inspector, which reveals the full value of the cell. This is especially useful when dealing with JSON values or log messages.

docs/docs/build/connectors/credentials.md

@@ -24,7 +24,7 @@ While Rill **can** infer credentials from your local environment (AWS CLI, Azure
 
 1. **Credentials referenced in connection strings or DSN within YAML files (RECOMMENDED)** - The UI creates YAML configurations that reference credentials from your `.env` file using templating (see [Connector YAML](/reference/project-files/connectors) for more details)
 2. **Credentials passed in as variables** - When starting Rill Developer via `rill start --env key=value` (see [templating](/build/connectors/templating) for more details)
-3. **Credentials configured via CLI** - For [AWS](/build/connectors/data-source/s3#local-aws-credentials-local-development-only) / [Azure](/build/connectors/data-source/azure#azure-cli-authentication-local-development-only) / [Google Cloud](/build/connectors/data-source/gcs#method-3-local-google-cloud-cli-credentials) - **NOT RECOMMENDED for production use**
+3. **Credentials configured via CLI** - For [AWS](/build/connectors/data-source/s3#method-4-local-aws-credentials-local-development-only) / [Azure](/build/connectors/data-source/azure#method-5-azure-cli-authentication-local-development-only) / [Google Cloud](/build/connectors/data-source/gcs#method-4-local-google-cloud-cli-credentials) - **NOT RECOMMENDED for production use**
 
 For more details, please refer to the corresponding [connector](/build/connectors) or [OLAP engine](/build/connectors/olap) page.
 

docs/docs/build/connectors/data-source/azure.md

@@ -14,79 +14,332 @@ sidebar_position: 05
 azure://<account>.blob.core.windows.net/<container>/path/file.csv
 ```
 
-## Connect to Azure Blob Storage
+## Authentication Methods
 
-To connect to Azure Blob Storage, you need to provide authentication credentials. You have four options:
+To connect to Azure Blob Storage, you can choose from four authentication options:
 
-1. **Use Storage Account Key** (recommended for cloud deployment)
-2. **Use Connection String** (alternative for cloud deployment)
-3. **Use Shared Access Signature (SAS) Token** (most secure, fine-grained control)
-4. **Use Azure CLI authentication** (local development only - not recommended for production)
+1. **Storage Account Key** (recommended for cloud deployment)
+2. **Connection String** (alternative for cloud deployment)
+3. **Shared Access Signature (SAS) Token** (most secure, fine-grained control)
+4. **Public** (for publicly accessible containers - no authentication required)
+5. **Azure CLI authentication** (local development only - not recommended for production)
 
-Choose the method that best fits your setup. For production deployments to Rill Cloud, use Storage Account Key, Connection String, or SAS tokens. Azure CLI authentication only works for local development and will cause deployment failures.
+:::tip Authentication Methods
+Choose the method that best fits your setup. For production deployments to Rill Cloud, use Storage Account Key, Connection String, or SAS tokens. Public containers don't require authentication and skip connector creation. Azure CLI authentication only works for local development and will cause deployment failures.
+:::
+
+## Using the Add Data UI
+
+When you add an Azure Blob Storage data model through the Rill UI, you'll see four authentication options:
+
+- **Storage Account Key**, **Connection String**, or **SAS Token**: The process follows two steps:
+  1. **Configure Authentication** - Set up your Azure connector with credentials
+  2. **Configure Data Model** - Define which container and objects to ingest
+  The UI will automatically create both the connector file and model file for you.
+
+- **Public**: For publicly accessible containers, you skip the connector creation step and go directly to:
+  1. **Configure Data Model** - Define which container and objects to ingest
+  The UI will only create the model file (no connector file is needed).
+
+:::note Manual Configuration Only
+Azure CLI authentication is only available through manual configuration. See [Method 5: Azure CLI Authentication](#method-5-azure-cli-authentication-local-development-only) for setup instructions.
+:::
+
+---
+
+## Method 1: Storage Account Key (Recommended)
+
+Storage Account Key credentials provide reliable authentication for Azure Blob Storage. This method works for both local development and Rill Cloud deployments.
+
+### Using the UI
+
+1. Click **Add Data** in your Rill project
+2. Select **Azure Blob Storage** as the data model type
+3. In the authentication step:
+   - Choose **Storage Account Key**
+   - Enter your Storage Account name
+   - Enter your Storage Account Key
+   - Name your connector (e.g., `my_azure`)
+4. In the data model configuration step:
+   - Enter your container name and object path
+   - Configure other model settings as needed
+5. Click **Create** to finalize
+
+The UI will automatically create both the connector file and model file for you.
+
+### Manual Configuration
 
-### Storage Account Key
+If you prefer to configure manually, create two files:
 
-To ensure seamless deployment to Rill Cloud, configure your Azure Storage Account Key directly in your project's `.env` file instead of relying solely on Azure CLI authentication (which only works locally).
+**Step 1: Create connector configuration**
+
+Create `connectors/my_azure.yaml`:
 
 ```yaml
 type: connector
-
 driver: azure
 
 azure_storage_account: rilltest
 azure_storage_key: "{{ .env.connector.azure.azure_storage_key }}"
 ```
 
-This approach ensures your Azure Blob Storage sources authenticate consistently across both local development and cloud deployment. Follow the [Azure Documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) to retrieve your storage account keys.
+**Step 2: Create model configuration**
+
+Create `models/my_azure_data.yaml`:
+
+```yaml
+type: model
+connector: duckdb
+
+sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/path/to/data/*.parquet')
+
+refresh:
+  cron: "0 */6 * * *"
+```
+
+**Step 3: Add credentials to `.env`**
+
+```bash
+connector.azure.azure_storage_key=your_storage_account_key
+```
+
+Follow the [Azure Documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) to retrieve your storage account keys.
+
+---
+
+## Method 2: Connection String
+
+Connection String provides an alternative authentication method for Azure Blob Storage.
+
+### Using the UI
 
-### Connection String
+1. Click **Add Data** in your Rill project
+2. Select **Azure Blob Storage** as the data model type
+3. In the authentication step:
+   - Choose **Connection String**
+   - Enter your Connection String
+   - Name your connector (e.g., `my_azure_conn`)
+4. In the data model configuration step:
+   - Enter your container name and object path
+   - Configure other model settings as needed
+5. Click **Create** to finalize
 
-To ensure seamless deployment to Rill Cloud, configure your Azure Blob Storage credentials using a connection string directly in your project's `.env` file instead of relying solely on Azure CLI authentication (which only works locally).
+### Manual Configuration
+
+**Step 1: Create connector configuration**
+
+Create `connectors/my_azure_conn.yaml`:
 
 ```yaml
 type: connector
-
 driver: azure
 
 azure_storage_connection_string: "{{ .env.connector.azure.azure_storage_connection_string }}"
 ```
 
-This approach ensures your Azure Blob Storage sources authenticate consistently across both local development and cloud deployment. Follow the [Azure Documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) to retrieve your connection string.
+**Step 2: Create model configuration**
 
-### Shared Access Signature (SAS) Token
+Create `models/my_azure_data.yaml`:
 
-Use Shared Access Signature (SAS) tokens as an alternative authentication method for Azure Blob Storage. SAS tokens provide fine-grained access control with specific permissions and expiration times for secure access to your storage resources.
+```yaml
+type: model
+connector: duckdb
+
+sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/path/to/data/*.parquet')
+
+refresh:
+  cron: "0 */6 * * *"
+```
+
+**Step 3: Add credentials to `.env`**
+
+```bash
+connector.azure.azure_storage_connection_string=your_connection_string
+```
+
+Follow the [Azure Documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) to retrieve your connection string.
+
+---
+
+## Method 3: Shared Access Signature (SAS) Token
+
+SAS tokens provide fine-grained access control with specific permissions and expiration times for secure access to your storage resources.
+
+### Using the UI
+
+1. Click **Add Data** in your Rill project
+2. Select **Azure Blob Storage** as the data model type
+3. In the authentication step:
+   - Choose **SAS Token**
+   - Enter your Storage Account name
+   - Enter your SAS Token
+   - Name your connector (e.g., `my_azure_sas`)
+4. In the data model configuration step:
+   - Enter your container name and object path
+   - Configure other model settings as needed
+5. Click **Create** to finalize
+
+### Manual Configuration
+
+**Step 1: Create connector configuration**
+
+Create `connectors/my_azure_sas.yaml`:
 
 ```yaml
 type: connector
-
 driver: azure
 
 azure_storage_account: rilltest 
 azure_storage_sas_token: "{{ .env.connector.azure.azure_storage_sas_token }}"
 ```
 
-This method provides fine-grained access control and enhanced security for your Azure Blob Storage connections. Follow the [Azure Documentation](https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens?tabs=Containers) to create your Azure SAS token.
+**Step 2: Create model configuration**
 
-###  Azure CLI Authentication (Local Development Only)
+Create `models/my_azure_data.yaml`:
 
-:::warning Not recommended for production
-Azure CLI authentication only works for local development. If you deploy to Rill Cloud using this method, your dashboards will fail. Use one of the methods above for production deployments.
-:::
+```yaml
+type: model
+connector: duckdb
 
-1. Install the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) if not already installed.
-2. Open a terminal window and run the following command to log in to your Azure account: `az login`
-3. Verify your authentication status: `az account show`
+sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/path/to/data/*.parquet')
 
-You've now configured Azure access from your local environment. Rill will automatically detect and use these credentials when you connect to Azure Blob Storage sources.
+refresh:
+  cron: "0 */6 * * *"
+```
 
-:::tip Cloud Credentials Management
+**Step 3: Add credentials to `.env`**
 
-If your project is already deployed to Rill Cloud with configured credentials, use `rill env pull` to [retrieve and sync these cloud credentials](/build/connectors/credentials/#rill-env-pull) to your local `.env` file. **Warning**: This operation will overwrite any existing local credentials for this source.
+```bash
+connector.azure.azure_storage_sas_token=your_sas_token
+```
+
+Follow the [Azure Documentation](https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens?tabs=Containers) to create your Azure SAS token.
+
+---
+
+## Method 4: Public Containers
+
+For publicly accessible Azure Blob Storage containers, you don't need to create a connector. Simply use the Azure URI directly in your model configuration.
+
+### Using the UI
+
+1. Click **Add Data** in your Rill project
+2. Select **Azure Blob Storage** as the data model type
+3. In the authentication step:
+   - Choose **Public**
+   - The UI will skip connector creation and proceed directly to data model configuration
+4. In the data model configuration step:
+   - Enter your container name and object path
+   - Configure other model settings as needed
+5. Click **Create** to finalize
+
+The UI will only create the model file (no connector file is created).
+
+### Manual Configuration
+
+For public containers, you only need to create a model file. No connector configuration is required.
+
+Create `models/my_azure_data.yaml`:
+
+```yaml
+type: model
+connector: duckdb
+
+sql: SELECT * FROM read_parquet('azure://publicaccount.blob.core.windows.net/my-public-container/path/to/data/*.parquet')
+
+refresh:
+  cron: "0 */6 * * *"
+```
+
+---
+
+## Method 5: Azure CLI Authentication (Local Development Only)
+
+For local development, you can use credentials from the Azure CLI. This method is **not suitable for production** or Rill Cloud deployments. This method is only available through manual configuration, and you don't need to create a connector file.
+
+### Setup
+
+1. Install the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) if not already installed
+2. Authenticate with your Azure account:
+   ```bash
+   az login
+   ```
+3. Verify your authentication status:
+   ```bash
+   az account show
+   ```
+4. Create your model file (no connector needed)
+
+### Model Configuration
+
+Create `models/my_azure_data.yaml`:
+
+```yaml
+type: model
+connector: duckdb
+
+sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/path/to/data/*.parquet')
 
+refresh:
+  cron: "0 */6 * * *"
+```
+
+Rill will automatically detect and use your local Azure CLI credentials when no connector is specified.
+
+:::warning
+This method only works for local development. Deploying to Rill Cloud with this configuration will fail because the cloud environment doesn't have access to your local credentials. Always use Storage Account Key, Connection String, or SAS tokens for production deployments.
 :::
 
+## Using Azure Blob Storage Data in Models
+
+Once your connector is configured (or for public containers, no connector needed), you can reference Azure Blob Storage paths in your model SQL queries using DuckDB's Azure functions.
+
+### Basic Example
+
+**With a connector (authenticated):**
+
+```yaml
+type: model
+connector: duckdb
+
+sql: SELECT * FROM read_parquet('azure://rilltest.blob.core.windows.net/my-container/data/*.parquet')
+
+refresh:
+  cron: "0 */6 * * *"
+```
+
+**Public container (no connector needed):**
+
+```yaml
+type: model
+connector: duckdb
+
+sql: SELECT * FROM read_parquet('azure://publicaccount.blob.core.windows.net/my-public-container/data/*.parquet')
+
+refresh:
+  cron: "0 */6 * * *"
+```
+
+### Path Patterns
+
+You can use wildcards to read multiple files:
+
+```sql
+-- Single file
+SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/file.parquet')
+
+-- All files in a directory
+SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/*.parquet')
+
+-- All files in nested directories
+SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/**/*.parquet')
+
+-- Files matching a pattern
+SELECT * FROM read_parquet('azure://account.blob.core.windows.net/container/data/2024-*.parquet')
+```
+
+---
+
 ## Deploy to Rill Cloud
 
 When deploying a project to Rill Cloud, Rill requires you to explicitly provide either an Azure Blob Storage connection string, Azure Storage Key, or Azure Storage SAS token for the containers used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#azure) for more information.
@@ -95,3 +348,4 @@ If you subsequently add sources that require new credentials (or if you simply e
 ```
 rill env push
 ```
+