Compile AI Documentation Bundle from GCS #733
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Compile AI Documentation Bundle from GCS | |
| on: | |
| # Nightly. The scheduled run is the only one that opens a bundle-refresh PR, | |
| # so the served download stays at most a day behind. Dispatch-driven runs | |
| # still rebuild GHCR, Artifact Registry, and Cloud Run on every source change. | |
| schedule: | |
| - cron: '0 2 * * *' | |
| repository_dispatch: | |
| types: [ai-docs-source-updated] | |
| workflow_dispatch: | |
| concurrency: | |
| group: compile-ai-docs-${{ github.ref }} | |
| cancel-in-progress: false | |
| permissions: {} | |
| jobs: | |
| compile-docs: | |
| if: github.repository == 'chainguard-dev/edu' | |
| runs-on: ubuntu-latest | |
| environment: documentation | |
| permissions: | |
| contents: read | |
| id-token: write # For GCP workload identity federation | |
| packages: write # For pushing to GHCR | |
| steps: | |
| - name: Harden Runner | |
| uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1 | |
| with: | |
| egress-policy: block | |
| allowed-endpoints: > | |
| api.github.com:443 | |
| *.pkg.dev:443 | |
| *.r2.cloudflarestorage.com:443 | |
| apk.cgr.dev:443 | |
| cgr.dev:443 | |
| dl.google.com:443 | |
| files.pythonhosted.org:443 | |
| fulcio.sigstore.dev:443 | |
| ghcr.io:443 | |
| github.com:443 | |
| iamcredentials.googleapis.com:443 | |
| oauth2.googleapis.com:443 | |
| objects.githubusercontent.com:443 | |
| release-assets.githubusercontent.com:443 | |
| pypi.org:443 | |
| raw.githubusercontent.com:443 | |
| rekor.sigstore.dev:443 | |
| run.googleapis.com:443 | |
| us-central1-run.googleapis.com:443 | |
| sts.googleapis.com:443 | |
| storage.googleapis.com:443 | |
| timestamp.sigstore.dev:443 | |
| tuf-repo-cdn.sigstore.dev:443 | |
| - name: Checkout edu repository | |
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| with: | |
| persist-credentials: false # No git write occurs in this workflow; outputs go to GCS/GHCR | |
| - name: Authenticate to Google Cloud | |
| uses: step-security/google-github-auth@775fc4c80760272ef389c9f9f8d98de7db0c170d # v3.0.2 | |
| with: | |
| workload_identity_provider: "projects/456977358484/locations/global/workloadIdentityPools/chainguard-academy/providers/chainguard-edu" | |
| service_account: "github-chainguard-academy@chainguard-academy.iam.gserviceaccount.com" | |
| - name: Set up Cloud SDK | |
| uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db # v3.0.1 | |
| - name: Set up Python | |
| uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | |
| with: | |
| python-version: '3.10' | |
| - name: Install dependencies | |
| run: | | |
| if [ -f scripts/requirements.txt ]; then | |
| pip install -r scripts/requirements.txt | |
| else | |
| pip install pyyaml | |
| fi | |
| - name: Download documentation from GCS | |
| run: | | |
| echo "Downloading documentation bundles from GCS..." | |
| # Create directories for extracted content | |
| mkdir -p ../images-private | |
| mkdir -p ../courses | |
| mkdir -p ../dfc | |
| # Validate tar archive has no path traversal before extracting | |
| validate_tar() { | |
| local archive="$1" | |
| if tar -tzf "$archive" | grep -qE '(^\./?\.\.|^/|\.\.)'; then | |
| echo "ERROR: Archive $archive contains path traversal entries" | |
| exit 1 | |
| fi | |
| } | |
| # Download and extract edu docs (from GCS, not local) | |
| if gcloud storage cp gs://academy-all-docs/edu/docs-export.tar.gz /tmp/edu.tar.gz \ | |
| --project=chainguard-academy 2>/dev/null; then | |
| echo "Extracting edu documentation..." | |
| validate_tar /tmp/edu.tar.gz | |
| # For edu, we'll extract to a temp location and copy selectively | |
| mkdir -p /tmp/edu-docs | |
| tar -xzf /tmp/edu.tar.gz -C /tmp/edu-docs --strip-components=1 | |
| # Copy content but avoid duplicating what's already in the local edu repo | |
| if [ -d "/tmp/edu-docs/content" ]; then | |
| echo "✓ edu docs downloaded from GCS ($(find /tmp/edu-docs -name "*.md" | wc -l) files)" | |
| fi | |
| else | |
| echo "⚠ edu docs not found in GCS (will use local content)" | |
| fi | |
| # Download and extract images-private docs | |
| if gcloud storage cp gs://academy-all-docs/images-private/docs-export.tar.gz /tmp/images-private.tar.gz \ | |
| --project=chainguard-academy 2>/dev/null; then | |
| echo "Extracting images-private documentation..." | |
| validate_tar /tmp/images-private.tar.gz | |
| tar -xzf /tmp/images-private.tar.gz -C ../images-private | |
| echo "✓ images-private docs downloaded" | |
| else | |
| echo "⚠ images-private docs not found in GCS (will use placeholder)" | |
| echo "# Images Private Documentation" > ../images-private/README.md | |
| echo "Documentation not yet available" >> ../images-private/README.md | |
| fi | |
| # Download and extract courses docs | |
| if gcloud storage cp gs://academy-all-docs/courses/docs-export.tar.gz /tmp/courses.tar.gz \ | |
| --project=chainguard-academy 2>/dev/null; then | |
| echo "Extracting courses documentation..." | |
| validate_tar /tmp/courses.tar.gz | |
| tar -xzf /tmp/courses.tar.gz -C ../courses --strip-components=1 | |
| echo "✓ courses docs downloaded" | |
| else | |
| echo "⚠ courses docs not found in GCS (will use placeholder)" | |
| echo "# Courses Documentation" > ../courses/README.md | |
| echo "Documentation not yet available" >> ../courses/README.md | |
| fi | |
| # Download and extract dfc docs (Dockerfile Converter mappings) | |
| if gcloud storage cp gs://academy-all-docs/dfc/docs-export.tar.gz /tmp/dfc.tar.gz \ | |
| --project=chainguard-academy 2>/dev/null; then | |
| echo "Extracting dfc documentation..." | |
| validate_tar /tmp/dfc.tar.gz | |
| tar -xzf /tmp/dfc.tar.gz -C ../dfc | |
| echo "✓ dfc docs downloaded" | |
| else | |
| echo "⚠ dfc docs not found in GCS (will skip dfc mappings)" | |
| fi | |
| # Download metadata files for logging | |
| echo "" | |
| echo "Metadata information:" | |
| for repo in edu images-private courses; do | |
| if gcloud storage cp gs://academy-all-docs/${repo}/metadata.json /tmp/${repo}-metadata.json \ | |
| --project=chainguard-academy 2>/dev/null; then | |
| echo "${repo}: $(cat /tmp/${repo}-metadata.json | jq -r '.export_time // "unknown"')" | |
| fi | |
| done | |
| - name: Compile documentation | |
| run: | | |
| echo "Compiling documentation bundle..." | |
| python3 scripts/compile_docs.py | |
| # Verify output | |
| if [ ! -f "static/downloads/chainguard-complete-docs.md" ]; then | |
| echo "Error: Documentation compilation failed" | |
| exit 1 | |
| fi | |
| # Catch a runaway compile before anything is published. | |
| size=$(stat -c%s static/downloads/chainguard-complete-docs.md) | |
| if [ "$size" -gt 52428800 ]; then | |
| echo "Error: bundle is $size bytes, over the 50MB ceiling" | |
| exit 1 | |
| fi | |
| echo "Documentation compiled successfully" | |
| ls -lh static/downloads/chainguard-complete-docs.md | |
| - name: Scan the bundle for credentials | |
| run: | | |
| # The bundle is published publicly, so check it before it leaves the | |
| # runner. compile_docs.py redacts the key formats it knows about; this | |
| # catches the ones it does not. Report line numbers only, so a | |
| # suspected key never lands in a public workflow log. | |
| bundle=static/downloads/chainguard-complete-docs.md | |
| pattern='(AKIA[0-9A-Z]{16}|ghp_[0-9a-zA-Z]{36}|ghs_[0-9a-zA-Z]{36}|sk-[0-9a-zA-Z]{48})' | |
| if grep -qE "$pattern" "$bundle"; then | |
| echo "Error: potential credentials in $bundle at these lines:" | |
| grep -nE "$pattern" "$bundle" | cut -d: -f1 | |
| exit 1 | |
| fi | |
| echo "No credential patterns found" | |
| - name: Generate image catalog | |
| env: | |
| COMMIT_SHA: ${{ github.sha }} | |
| run: | | |
| python3 scripts/generate_image_catalog.py \ | |
| --docs static/downloads/chainguard-complete-docs.md \ | |
| --mappings data/package-mappings.yaml \ | |
| --output scripts/image-catalog.json \ | |
| --commit "$COMMIT_SHA" | |
| - name: Create AI-specific bundle | |
| run: | | |
| cd static/downloads | |
| # Rename for clarity | |
| cp chainguard-complete-docs.md chainguard-ai-docs.md | |
| # Copy image catalog into downloads | |
| cp ../../scripts/image-catalog.json . | |
| # Create compressed versions (include catalog in bundles) | |
| tar -czf chainguard-ai-docs.tar.gz chainguard-ai-docs.md image-catalog.json | |
| gzip -k chainguard-ai-docs.md | |
| zip chainguard-ai-docs.zip chainguard-ai-docs.md image-catalog.json | |
| # Generate checksums | |
| sha256sum chainguard-ai-docs.md image-catalog.json > checksums.txt | |
| echo "AI documentation bundle created:" | |
| ls -lh chainguard-ai-docs.* | |
| - name: Install cosign | |
| if: github.ref == 'refs/heads/main' | |
| uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 | |
| - name: Login to GHCR | |
| if: github.ref == 'refs/heads/main' | |
| uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 | |
| with: | |
| registry: ghcr.io | |
| username: ${{ github.actor }} | |
| password: ${{ secrets.GITHUB_TOKEN }} | |
| - name: Build and push container image | |
| if: github.ref == 'refs/heads/main' | |
| env: | |
| REPO_OWNER: ${{ github.repository_owner }} | |
| COMMIT_SHA: ${{ github.sha }} | |
| run: | | |
| # Copy files for container build | |
| cp static/downloads/chainguard-ai-docs.md scripts/ | |
| # image-catalog.json is already in scripts/ from the generate step | |
| # Create container-specific checksums (only for files included in container) | |
| cd scripts | |
| sha256sum chainguard-ai-docs.md image-catalog.json > checksums.txt | |
| cd .. | |
| # Build container with GitHub Container Registry | |
| docker build -f scripts/Dockerfile.ai-docs -t "ghcr.io/$REPO_OWNER/ai-docs:latest" scripts/ | |
| docker tag "ghcr.io/$REPO_OWNER/ai-docs:latest" "ghcr.io/$REPO_OWNER/ai-docs:$COMMIT_SHA" | |
| # Push images and capture digest for signing | |
| PUSH_OUTPUT=$(docker push "ghcr.io/$REPO_OWNER/ai-docs:latest") | |
| echo "$PUSH_OUTPUT" | |
| # Extract digest more reliably - handle both formats | |
| DIGEST=$(echo "$PUSH_OUTPUT" | grep -oP 'sha256:[a-f0-9]{64}' | tail -1) | |
| if [ -z "$DIGEST" ]; then | |
| # Fallback: use inspect to get the digest | |
| DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "ghcr.io/$REPO_OWNER/ai-docs:latest" | grep -oP 'sha256:[a-f0-9]{64}') | |
| fi | |
| if [ -z "$DIGEST" ]; then | |
| echo "ERROR: Failed to capture image digest from docker push" | |
| exit 1 | |
| fi | |
| echo "Captured digest: $DIGEST" | |
| docker push "ghcr.io/$REPO_OWNER/ai-docs:$COMMIT_SHA" || { echo "ERROR: Failed to push SHA-tagged image"; exit 1; } | |
| # Sign container image by immutable digest | |
| cosign sign --yes "ghcr.io/$REPO_OWNER/ai-docs@$DIGEST" | |
| - name: Smoke-test MCP server image | |
| if: github.ref == 'refs/heads/main' | |
| env: | |
| REPO_OWNER: ${{ github.repository_owner }} | |
| run: | | |
| # Gate the deploy: confirm the freshly built image actually boots and | |
| # serves the MCP HTTP transport before it reaches Cloud Run (DOCS-91). | |
| scripts/smoke-test-mcp.sh "ghcr.io/$REPO_OWNER/ai-docs:latest" | |
| - name: Push container image to Artifact Registry | |
| if: github.ref == 'refs/heads/main' | |
| env: | |
| REGISTRY_URL: ${{ secrets.REGISTRY_URL }} | |
| PROJECT_ID: ${{ secrets.PROJECT_ID }} | |
| REPOSITORY: ${{ secrets.REPOSITORY }} | |
| COMMIT_SHA: ${{ github.sha }} | |
| REPO_OWNER: ${{ github.repository_owner }} | |
| run: | | |
| gcloud auth configure-docker "${REGISTRY_URL%%/*}" --quiet | |
| AR_IMAGE="$REGISTRY_URL/$PROJECT_ID/$REPOSITORY/ai-docs" | |
| docker tag "ghcr.io/${REPO_OWNER}/ai-docs:$COMMIT_SHA" "$AR_IMAGE:$COMMIT_SHA" | |
| docker push "$AR_IMAGE:$COMMIT_SHA" || { echo "ERROR: Failed to push to Artifact Registry"; exit 1; } | |
| - name: Deploy MCP server to Cloud Run | |
| if: github.ref == 'refs/heads/main' | |
| env: | |
| REGISTRY_URL: ${{ secrets.REGISTRY_URL }} | |
| PROJECT_ID: ${{ secrets.PROJECT_ID }} | |
| REPOSITORY: ${{ secrets.REPOSITORY }} | |
| COMMIT_SHA: ${{ github.sha }} | |
| run: | | |
| gcloud run services update mcp-server \ | |
| --image "$REGISTRY_URL/$PROJECT_ID/$REPOSITORY/ai-docs:$COMMIT_SHA" \ | |
| --command "/usr/local/bin/serve-mcp-http" \ | |
| --region us-central1 \ | |
| --project "$PROJECT_ID" | |
| # Allow unauthenticated access (CI terraform SA lacks setIamPolicy) | |
| gcloud run services add-iam-policy-binding mcp-server \ | |
| --region us-central1 \ | |
| --member "allUsers" \ | |
| --role "roles/run.invoker" \ | |
| --project "$PROJECT_ID" || true | |
| - name: Sign bundle files | |
| if: github.ref == 'refs/heads/main' | |
| run: | | |
| cd static/downloads | |
| cosign sign-blob chainguard-ai-docs.tar.gz \ | |
| --yes \ | |
| --bundle=chainguard-ai-docs.tar.gz.bundle | |
| cosign sign-blob chainguard-ai-docs.md \ | |
| --yes \ | |
| --bundle=chainguard-ai-docs.md.bundle | |
| echo "Bundle files signed" | |
| - name: Upload bundle back to GCS for distribution | |
| env: | |
| COMMIT_SHA: ${{ github.sha }} | |
| EVENT_NAME: ${{ github.event_name }} | |
| run: | | |
| echo "Uploading compiled bundle to GCS..." | |
| # Upload the compiled bundle for easy distribution | |
| gcloud storage cp static/downloads/chainguard-ai-docs.tar.gz \ | |
| gs://academy-all-docs/compiled/chainguard-ai-docs.tar.gz \ | |
| --project=chainguard-academy | |
| # Upload checksums | |
| gcloud storage cp static/downloads/checksums.txt \ | |
| gs://academy-all-docs/compiled/checksums.txt \ | |
| --project=chainguard-academy | |
| # Upload cosign bundle files for artifact verification (only present on main branch builds) | |
| if [ -f "static/downloads/chainguard-ai-docs.tar.gz.bundle" ]; then | |
| gcloud storage cp static/downloads/chainguard-ai-docs.tar.gz.bundle \ | |
| gs://academy-all-docs/compiled/chainguard-ai-docs.tar.gz.bundle \ | |
| --project=chainguard-academy | |
| gcloud storage cp static/downloads/chainguard-ai-docs.md.bundle \ | |
| gs://academy-all-docs/compiled/chainguard-ai-docs.md.bundle \ | |
| --project=chainguard-academy | |
| fi | |
| # Create and upload compilation metadata | |
| python3 -c " | |
| import json, datetime, os | |
| meta = { | |
| 'compiled_at': datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ'), | |
| 'commit': os.environ['COMMIT_SHA'], | |
| 'triggered_by': os.environ['EVENT_NAME'], | |
| 'file_size': str(os.path.getsize('static/downloads/chainguard-ai-docs.md')) | |
| } | |
| with open('/tmp/compilation-metadata.json', 'w') as f: | |
| json.dump(meta, f, indent=2) | |
| " | |
| gcloud storage cp /tmp/compilation-metadata.json \ | |
| gs://academy-all-docs/compiled/metadata.json \ | |
| --project=chainguard-academy | |
| echo "✓ Bundle uploaded to GCS" | |
| # Hand the bundle to the publish job rather than opening the pull request | |
| # here. That job needs a git-write token, and this job builds and signs | |
| # container images -- keep the two credential scopes apart. | |
| - name: Upload the bundle for the publish job | |
| if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | |
| with: | |
| name: compiled-bundle | |
| path: static/downloads/chainguard-complete-docs.md | |
| retention-days: 1 | |
| # Refresh the copy of the bundle that edu.chainguard.dev actually serves. | |
| # Hugo publishes static/downloads/ straight from git, so the served download | |
| # only changes when a commit lands -- see DOCS-158. | |
| publish-bundle: | |
| if: >- | |
| github.repository == 'chainguard-dev/edu' && | |
| github.ref == 'refs/heads/main' && | |
| (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') | |
| needs: compile-docs | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read | |
| id-token: write # Federate a token via octo-sts | |
| env: | |
| # Single source of truth: the preflight check and the | |
| # create-pull-request input must never drift apart. | |
| PR_BRANCH: bot/docs-bundle-refresh | |
| steps: | |
| - name: Harden Runner | |
| uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1 | |
| with: | |
| egress-policy: block | |
| allowed-endpoints: > | |
| *.githubapp.com:443 | |
| api.github.com:443 | |
| edu.chainguard.dev:443 | |
| github.com:443 | |
| objects.githubusercontent.com:443 | |
| octo-sts.dev:443 | |
| release-assets.githubusercontent.com:443 | |
| - uses: octo-sts/action@f603d3be9d8dd9871a265776e625a27b00effe05 # v1.1.1 | |
| id: octo-sts | |
| with: | |
| scope: ${{ github.repository }} | |
| identity: ai-docs | |
| - name: Checkout edu repository | |
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| # zizmor: ignore[artipacked] - credentials needed for create-pull-request to push | |
| with: | |
| token: ${{ steps.octo-sts.outputs.token }} | |
| persist-credentials: true | |
| - name: Download the compiled bundle | |
| uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | |
| with: | |
| name: compiled-bundle | |
| path: static/downloads | |
| - name: Skip publishing when only the timestamp moved | |
| run: | | |
| # The bundle embeds a "_Compiled on:_" line that changes on every run, | |
| # so a plain diff is never empty. Compare without that line, and put | |
| # the committed copy back when nothing else moved -- then the step | |
| # below finds no change and opens no pull request. | |
| published=static/downloads/chainguard-complete-docs.md | |
| git show "HEAD:$published" > /tmp/published-bundle.md | |
| if diff -q <(grep -v '^_Compiled on: ' /tmp/published-bundle.md) \ | |
| <(grep -v '^_Compiled on: ' "$published") > /dev/null; then | |
| echo "Bundle content is unchanged; restoring the committed copy." | |
| cp /tmp/published-bundle.md "$published" | |
| else | |
| echo "Bundle content changed; opening a pull request." | |
| fi | |
| - name: Fail early if a tag shadows the pull request branch | |
| run: | | |
| # create-pull-request fetches "$PR_BRANCH:refs/remotes/origin/$PR_BRANCH", | |
| # and a bare ref name resolves against tags as well as heads. A tag of | |
| # the same name therefore makes the action believe the branch already | |
| # exists, and the push then fails with git's opaque "stale info" -- the | |
| # DOCS-158 failure. Every tag in this repository starts with "ai-docs" | |
| # and a "Block Tag Deletion" ruleset makes them permanent, so name the | |
| # problem here rather than leaving the next person to decode it. | |
| if git ls-remote --exit-code origin "refs/tags/$PR_BRANCH" > /dev/null 2>&1; then | |
| echo "Error: a tag named '$PR_BRANCH' exists and shadows the branch" | |
| echo "this job pushes to. Set PR_BRANCH to a name no tag uses, and" | |
| echo "keep it outside the ai-docs* namespace." | |
| exit 1 | |
| fi | |
| echo "No tag shadows '$PR_BRANCH'." | |
| - name: Open a pull request for the refreshed bundle | |
| uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1 | |
| with: | |
| token: ${{ steps.octo-sts.outputs.token }} | |
| # Only the served bundle. Everything else the compile job writes to | |
| # static/downloads is a build artifact that belongs in GHCR and GCS. | |
| add-paths: static/downloads/chainguard-complete-docs.md | |
| branch: ${{ env.PR_BRANCH }} | |
| commit-message: Refresh the compiled AI documentation bundle | |
| title: "[AI Docs] Refresh the documentation bundle" | |
| body: | | |
| Recompiled `static/downloads/chainguard-complete-docs.md`, the bundle | |
| served at | |
| <https://edu.chainguard.dev/downloads/chainguard-complete-docs.md>. | |
| Opened only when the compiled content changed, ignoring the embedded | |
| `_Compiled on:_` timestamp. Merging this is what refreshes the public | |
| download. | |
| labels: | | |
| documentation | |
| automated | |
| assignees: matthewhelmke | |
| - name: Warn when the served bundle has gone stale | |
| run: | | |
| # A pull request that nobody merges leaves the download stale, which is | |
| # how it went five months without a refresh (DOCS-158). Fail loudly | |
| # after two weeks of grace. | |
| served_date=$(curl -sS https://edu.chainguard.dev/downloads/chainguard-complete-docs.md \ | |
| | grep -m1 '^_Compiled on: ' | sed -E 's/^_Compiled on: ([0-9-]+).*/\1/') | |
| if [ -z "$served_date" ]; then | |
| echo "Error: could not read the served bundle's compile date" | |
| exit 1 | |
| fi | |
| age_days=$(( ( $(date -u +%s) - $(date -u -d "$served_date" +%s) ) / 86400 )) | |
| echo "Served bundle compiled $served_date ($age_days days ago)" | |
| if [ "$age_days" -gt 14 ]; then | |
| echo "Error: the served bundle is $age_days days old." | |
| echo "A refresh pull request is probably open and unmerged." | |
| exit 1 | |
| fi |