refactor: use GitHub Actions artifacts instead of base64 payload for documentation transfer

Co-authored-by: neilime <314088+neilime@users.noreply.github.com>

Author: Copilot

.github/README.md

@@ -94,15 +94,14 @@ The dispatcher workflow accepts these inputs:
 1. **Checkout** source repository
 2. **Collect** documentation files from `docs_path` and optionally `README.md`
 3. **Add frontmatter** with source metadata to each file
-4. **Bundle** documentation into compressed tar.gz
-5. **Encode** bundle as base64
-6. **Send** repository_dispatch event to public-docs with the encoded bundle
+4. **Upload** documentation as GitHub Actions artifact
+5. **Send** repository_dispatch event to public-docs with artifact metadata (name, run_id, repository)
 
 ## How the Receiver Works
 
 1. **Listen** for `repository_dispatch` events with type `sync-documentation`
-2. **Validate** required inputs (source_repo, target_path, documentation)
-3. **Decode** and extract documentation bundle
+2. **Validate** required inputs (source_repo, target_path, artifact_name, run_id, repository)
+3. **Download** artifact from the source repository using the provided metadata
 4. **Inject** documentation into `application/docs/{target_path}`
 5. **Build** documentation site to validate
 6. **If build succeeds**: Commit changes and deploy to GitHub Pages

.github/workflows/sync-docs-dispatcher.yml

@@ -50,10 +50,9 @@ jobs:
           
           echo "📦 Preparing documentation bundle from ${{ inputs.source_repo }}"
           
-          # Create temporary directory for documentation
-          TEMP_DIR=$(mktemp -d)
-          DOCS_DIR="$TEMP_DIR/docs"
-          mkdir -p "$DOCS_DIR"
+          # Create artifact directory
+          ARTIFACT_DIR="documentation-artifact"
+          mkdir -p "$ARTIFACT_DIR"
           
           # Function to add frontmatter
           add_frontmatter() {
@@ -91,8 +90,8 @@ jobs:
           # Copy README if requested
           if [ "${{ inputs.include_readme }}" = "true" ] && [ -f "README.md" ]; then
             echo "📄 Including README.md"
-            cp "README.md" "$DOCS_DIR/README.md"
-            add_frontmatter "$DOCS_DIR/README.md" "README.md"
+            cp "README.md" "$ARTIFACT_DIR/README.md"
+            add_frontmatter "$ARTIFACT_DIR/README.md" "README.md"
           fi
           
           # Copy documentation directory if it exists
@@ -102,7 +101,7 @@ jobs:
             # Find and copy all markdown files
             find "${{ inputs.docs_path }}" -type f \( -name "*.md" -o -name "*.mdx" \) | while read file; do
               rel_path="${file#${{ inputs.docs_path }}/}"
-              target_file="$DOCS_DIR/$rel_path"
+              target_file="$ARTIFACT_DIR/$rel_path"
               
               # Create directory structure
               mkdir -p "$(dirname "$target_file")"
@@ -121,7 +120,7 @@ jobs:
           fi
           
           # Create index file
-          cat > "$DOCS_DIR/_index.md" << EOF
+          cat > "$ARTIFACT_DIR/_index.md" << 'EOF'
           ---
           title: ${{ inputs.source_repo }}
           description: Documentation for ${{ inputs.source_repo }}
@@ -136,23 +135,21 @@ jobs:
           **Last Synced:** $(date -u +"%Y-%m-%dT%H:%M:%SZ")
           EOF
           
-          # Create compressed archive and encode in base64
-          cd "$TEMP_DIR"
-          tar -czf docs.tar.gz -C docs .
-          ENCODED=$(base64 -w 0 docs.tar.gz)
-          
-          # Save to output (GitHub Actions has a limit, so we need to handle large content)
-          echo "encoded_docs=$ENCODED" >> $GITHUB_OUTPUT
-          
           # Show summary
           echo "📊 Documentation bundle prepared:"
-          find docs -type f | while read file; do
-            echo "  - ${file#docs/}"
+          find "$ARTIFACT_DIR" -type f | while read file; do
+            echo "  - ${file#$ARTIFACT_DIR/}"
           done
           
-          # Cleanup
-          cd -
-          rm -rf "$TEMP_DIR"
+          # Save artifact name for later use
+          echo "artifact_name=docs-${{ inputs.source_repo }}-${{ github.run_id }}" >> $GITHUB_OUTPUT
+          
+      - name: Upload documentation artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ steps.prepare.outputs.artifact_name }}
+          path: documentation-artifact/
+          retention-days: 1
 
       - name: Dispatch to public-docs
         uses: peter-evans/repository-dispatch@v3
@@ -165,7 +162,9 @@ jobs:
               "source_repo": "${{ inputs.source_repo }}",
               "target_path": "${{ inputs.target_path }}",
               "branch": "${{ inputs.branch }}",
-              "documentation": "${{ steps.prepare.outputs.encoded_docs }}"
+              "artifact_name": "${{ steps.prepare.outputs.artifact_name }}",
+              "run_id": "${{ github.run_id }}",
+              "repository": "${{ github.repository }}"
             }
             
       - name: Summary
@@ -177,6 +176,7 @@ jobs:
           echo "- **Docs Path**: ${{ inputs.docs_path }}" >> $GITHUB_STEP_SUMMARY
           echo "- **Target Path**: ${{ inputs.target_path }}" >> $GITHUB_STEP_SUMMARY
           echo "- **Include README**: ${{ inputs.include_readme }}" >> $GITHUB_STEP_SUMMARY
-          echo "- **Status**: ✅ Dispatched to public-docs" >> $GITHUB_STEP_SUMMARY
+          echo "- **Artifact**: ${{ steps.prepare.outputs.artifact_name }}" >> $GITHUB_STEP_SUMMARY
+          echo "- **Status**: ✅ Artifact uploaded and dispatched to public-docs" >> $GITHUB_STEP_SUMMARY
           echo "" >> $GITHUB_STEP_SUMMARY
-          echo "The public-docs repository will process this documentation and publish it to the portal." >> $GITHUB_STEP_SUMMARY
+          echo "The public-docs repository will download the artifact and publish the documentation to the portal." >> $GITHUB_STEP_SUMMARY

.github/workflows/sync-docs-receiver.yml

@@ -43,14 +43,36 @@ jobs:
             exit 1
           fi
           
-          if [ -z "${{ github.event.client_payload.documentation }}" ]; then
-            echo "❌ Error: documentation content is required"
+          if [ -z "${{ github.event.client_payload.artifact_name }}" ]; then
+            echo "❌ Error: artifact_name is required"
+            exit 1
+          fi
+          
+          if [ -z "${{ github.event.client_payload.run_id }}" ]; then
+            echo "❌ Error: run_id is required"
+            exit 1
+          fi
+          
+          if [ -z "${{ github.event.client_payload.repository }}" ]; then
+            echo "❌ Error: repository is required"
             exit 1
           fi
           
           echo "✅ Input validation passed"
           echo "source_repo=${{ github.event.client_payload.source_repo }}" >> $GITHUB_OUTPUT
           echo "target_path=${{ github.event.client_payload.target_path }}" >> $GITHUB_OUTPUT
+          echo "artifact_name=${{ github.event.client_payload.artifact_name }}" >> $GITHUB_OUTPUT
+          echo "run_id=${{ github.event.client_payload.run_id }}" >> $GITHUB_OUTPUT
+          echo "repository=${{ github.event.client_payload.repository }}" >> $GITHUB_OUTPUT
+          
+      - name: Download documentation artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: ${{ steps.validate.outputs.artifact_name }}
+          path: documentation-download
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          repository: ${{ steps.validate.outputs.repository }}
+          run-id: ${{ steps.validate.outputs.run_id }}
 
       - name: Inject documentation
         run: |
@@ -66,14 +88,18 @@ jobs:
           TARGET_DIR="application/docs/$TARGET_PATH"
           mkdir -p "$TARGET_DIR"
           
-          # Decode and write documentation files
-          echo "${{ github.event.client_payload.documentation }}" | base64 -d | tar -xz -C "$TARGET_DIR"
-          
-          echo "✅ Documentation injected into $TARGET_DIR"
+          # Copy documentation files from downloaded artifact
+          if [ -d "documentation-download" ]; then
+            cp -r documentation-download/* "$TARGET_DIR/"
+            echo "✅ Documentation injected into $TARGET_DIR"
+          else
+            echo "❌ Error: Downloaded artifact directory not found"
+            exit 1
+          fi
           
           # List injected files
           echo "📁 Injected files:"
-          find "$TARGET_DIR" -type f -name "*.md" -o -name "*.mdx" | while read file; do
+          find "$TARGET_DIR" -type f \( -name "*.md" -o -name "*.mdx" \) | while read file; do
             echo "  - ${file#application/docs/}"
           done
           
@@ -133,6 +159,7 @@ jobs:
           echo "- **Source Repository**: ${{ github.event.client_payload.source_repo }}" >> $GITHUB_STEP_SUMMARY
           echo "- **Branch**: ${{ github.event.client_payload.branch || 'main' }}" >> $GITHUB_STEP_SUMMARY
           echo "- **Target Path**: ${{ github.event.client_payload.target_path }}" >> $GITHUB_STEP_SUMMARY
+          echo "- **Artifact**: ${{ github.event.client_payload.artifact_name }}" >> $GITHUB_STEP_SUMMARY
           echo "- **Changes**: ${{ steps.changes.outputs.changed == 'true' && '✅ Committed and deployed' || '📄 No changes' }}" >> $GITHUB_STEP_SUMMARY
           echo "" >> $GITHUB_STEP_SUMMARY
           echo "🌐 **Documentation Portal**: [docs.hoverkraft.cloud](https://docs.hoverkraft.cloud)" >> $GITHUB_STEP_SUMMARY

application/docs/documentation-system.md

@@ -28,16 +28,16 @@ The reusable dispatcher workflow (`sync-docs-dispatcher.yml`) that projects call
 │  │ 1. On commit to docs/ or README.md                 │ │
 │  │ 2. Collects documentation files                    │ │
 │  │ 3. Adds source metadata frontmatter                │ │
-│  │ 4. Creates compressed bundle (base64)              │ │
-│  │ 5. Sends repository_dispatch event                 │ │
+│  │ 4. Uploads documentation as artifact               │ │
+│  │ 5. Sends repository_dispatch event with metadata   │ │
 │  └────────────────────────────────────────────────────┘ │
 └───────────────┬─────────────────────────────────────────┘
-                │ repository_dispatch event
+                │ repository_dispatch event (with artifact ref)
                 ↓
 ┌─────────────────────────────────────────────────────────┐
 │  Documentation Portal (public-docs)                     │
 │  ┌──────────────────────────────────────────────────┐  │
-│  │ 6. Receives documentation bundle                  │  │
+│  │ 6. Downloads artifact from source repository     │  │
 │  │ 7. Validates and injects into docs/               │  │
 │  │ 8. Builds documentation site                      │  │
 │  │ 9. If build succeeds, commits and deploys         │  │
@@ -49,17 +49,19 @@ The reusable dispatcher workflow (`sync-docs-dispatcher.yml`) that projects call
 
 The receiver workflow (`sync-docs-receiver.yml`) in public-docs:
 - Listens for `repository_dispatch` events with type `sync-documentation`
-- Receives documentation bundle in the event payload
+- Downloads the documentation artifact from the source repository
 - Extracts and injects documentation into the appropriate location
 - Builds the documentation site to validate
 - If successful, commits changes and deploys to GitHub Pages
 
 **Advantages:**
 - Real-time updates when documentation changes
+- No size limitations (artifacts can handle large documentation bundles)
 - Validation before publishing (build must succeed)
 - No direct write access to public-docs needed for project repos
-- Token only needs `repository_dispatch` permission
+- Token only needs `repo` scope for repository_dispatch and artifact access
 - Failed builds don't corrupt the documentation
+- Supports image-heavy documentation without payload size constraints
 
 ### Setup