Implement Azure Storage Upload for Policy Docs & Assets

[DakshithaS]

Description

Enhance the policy creation flow to upload documentation and assets directly to Azure Blob Storage for improved scalability, reduced database load, and CDN-ready global content delivery.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Possible Duplicates

• Implement Azure Blob Storage Integration with Checksum Support for Policy Creation Flow #502

👤 Suggested Assignees

• DakshithaS

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[DakshithaS]

Proposed Policy Creation Flow with Implement Azure Storage Upload for Policy Docs & Assets

This outlines the complete policy creation flow, including uploading docs and assets to Azure Storage, as discussed.

API Contract

Endpoint

• Method: POST
• Path: /policies/{name}/versions/{version}
• Content-Type: multipart/form-data

Path Parameters

• name (string): Policy name (e.g., "rate-limiter")
• version (string): Version (e.g., "1.0.0")

Request Body (Multipart Form)

Required Fields

• policyName (string): Policy name (must match URL param)
• version (string): Version (must match URL param)
• metadata (JSON file/string): Policy metadata

  {
    "displayName": "Rate Limiter",
    "provider": "WSO2",
    "description": "Limits API requests",
    "categories": ["security"],
    "tags": ["rate", "limit"],
    "supportedPlatforms": ["APIM"]
  }

• artifact (file): ZIP file containing policy source code (max 50MB)
• definition (file): YAML file for policy definition (max 1MB)

Optional Fields

• docs[overview] (file): Markdown file for overview doc (max 10MB)
• docs[configuration] (file): Markdown file for configuration doc (max 10MB)
• docs[examples] (file): Markdown file for examples doc (max 10MB)
• docs[faq] (file): Markdown file for FAQ doc (max 10MB)
• assets[logo] (file): Logo image file (PNG/JPG, max 5MB)
• assets[banner] (file): Banner image file (PNG/JPG, max 5MB)
• assets[images/diagram] (file): Image files for docs (PNG/JPG/SVG, max 5MB each)

Note: Total request size must not exceed 100MB.

Response

• Success (200):

  {
    "success": true,
    "data": {
      "policyName": "rate-limiter",
      "version": "1.0.0",
      "status": "synced"
    }
  }

• Error (4xx/5xx):

  {
    "success": false,
    "error": {
      "code": "VALIDATION_ERROR",
      "message": "Invalid version format",
      "details": {...}
    }
  }

Image Handling Rules for docs

1. Relative Paths: Images referenced with relative paths (e.g., ![diagram](images/arch.png)) will be processed:

   • If a matching asset file is uploaded (e.g., assets[images/arch.png]), it will be uploaded to Azure and the markdown path will be rewritten to the Azure URL.
   • If the asset file is missing, log an error but continue processing the doc (leave the original path as-is).

2. Absolute URLs: Images with absolute URLs (e.g., ![diagram](https://external.com/image.png)) are left as-is. No fetching or uploading occurs.

3. Image Deduplication: Simple path-based deduplication:

   • If multiple docs reference the same image path (e.g., images/logo.png) and a single asset file is uploaded, it's uploaded once to Azure and all references use the same URL.
   • If different asset files are uploaded with the same filename, they are treated as separate assets (no content-based comparison).

Azure Storage Structure

policies/
  {policyName}/
    {version}/
      {policyName}-{version}.zip          # Policy artifact
      policy-definition.yaml               # Policy definition
      docs/
        overview.md                        # Documentation files
        configuration.md
        examples.md
        faq.md
      assets/
        logo.png                           # Logo asset
        banner.png                         # Banner asset
        images/
          diagram.png                      # Images referenced in docs
          architecture.svg

Sequence Diagram

sequenceDiagram
    participant Client
    participant SyncHandler
    participant SyncService
    participant AzureStorage
    participant Database

    Client->>SyncHandler: POST /policies/{name}/versions/{version} (multipart: policyName, version, metadata, artifact.zip, definition.yaml, docs[], assets[])
    SyncHandler->>SyncHandler: Validate request (names, version, file types, sizes)
    SyncHandler->>SyncService: Call SyncPolicy with SyncRequest
    SyncService->>SyncService: Check if policy version exists
    alt Version exists
        SyncService-->>SyncHandler: Return conflict error
        SyncHandler-->>Client: Error response
    else
        SyncService->>SyncService: Validate ZIP file and definition YAML
        SyncService->>SyncService: Generate checksum for artifact
        SyncService->>SyncService: Process docs: parse markdown, collect image refs (relative paths only)
        SyncService->>SyncService: Build image upload map (deduplicate by path)
        SyncService->>SyncService: Prepare all uploads (artifact, definition, docs, images, assets)
        
        par Parallel Upload Phase
            SyncService->>AzureStorage: Upload artifact.zip
            AzureStorage-->>SyncService: Return artifact URL
        and
            SyncService->>AzureStorage: Upload policy-definition.yaml
            AzureStorage-->>SyncService: Return definition URL
        and
            loop For each unique image (if file exists)
                SyncService->>AzureStorage: Upload image to assets/images/{path}
                AzureStorage-->>SyncService: Return image URL
            end
        and
            loop For each asset (logo, banner)
                SyncService->>AzureStorage: Upload asset to assets/{filename}
                AzureStorage-->>SyncService: Return asset URL
            end
        end
        
        Note over SyncService: Wait for all uploads to complete
        SyncService->>SyncService: Check for upload errors (fail if critical uploads failed)
        SyncService->>SyncService: Rewrite markdown (replace relative paths with Azure URLs, keep absolute URLs)
        
        par Upload Processed Docs
            loop For each doc
                SyncService->>AzureStorage: Upload rewritten doc.md to docs/{page}.md
                AzureStorage-->>SyncService: Return doc URL
            end
        end
        
        SyncService->>SyncService: Collect all Azure URLs, log any missing images
        SyncService->>SyncService: Update PolicyVersion with Azure URLs (logo, banner, etc.)
        SyncService->>Database: Create PolicyVersion record
        SyncService->>Database: Create PolicyDoc records with Azure URLs
        SyncService-->>SyncHandler: Return SyncResult (success)
        SyncHandler-->>Client: Success response with policy details
    end

Loading

Edge Cases & Error Handling

Handled Scenarios

1. Missing Optional Docs/Assets: Sync continues successfully even if no docs or assets are uploaded. Only artifact + definition are required.

2. Missing Image Files: If a doc references an image (e.g., images/logo.png) but the asset file is not uploaded:

   • Log error: "Image referenced but not found: images/logo.png"
   • Keep original markdown path unchanged
   • Continue processing the doc

3. External Image URLs: Absolute URLs (e.g., https://cdn.example.com/image.png) in markdown are preserved as-is. No fetching or modification.

4. Duplicate Image References: If multiple docs reference the same path (e.g., images/shared-logo.png):

   • Upload the image once to Azure
   • Rewrite all references to use the same Azure URL

5. Invalid YAML Definition: If policy-definition.yaml is malformed, return validation error before any uploads.

6. Version Conflict: If policy version already exists, return 409 Conflict error immediately (no uploads occur).

7. File Size Violations: If any file exceeds size limits, return 413 Payload Too Large error before processing.

8. Azure Upload Failures:

   • Critical uploads (artifact, definition): Retry up to 3 times with exponential backoff (1s, 2s, 4s). If all retries fail, abort sync and return error.
   • Optional uploads (docs/assets): Retry up to 2 times. If all retries fail, log warnings and continue (don't fail the entire sync).

9. Special Characters in Filenames: Sanitize asset filenames by replacing special characters with hyphens or underscores.

10. Concurrent Uploads: Database unique constraint on (policy_name, version) prevents duplicate versions. First upload succeeds, subsequent attempts get conflict errors.