Proposal: Automating content sync and delivery for the Actor Whitepaper Astro Site

[vancura]

Below is an overview and proposal for merging and automating the content from this repository to the Astro-based site currently in another repository: @vancura/developer-actor. The goal is to create a single source of truth for the Whitepaper, ensure that content is published cleanly to the Astro site, and automate the process using GitHub Actions or a similar CI/CD approach.

Background and current setup

1. The actor-whitepaper content (this repo) is currently manually copied into the Astro-based site. This repetitive process is prone to errors and omissions.
2. The Astro site code lives in a separate repo, @vancura/developer-actor.
3. The original @apify/actor-whitepaper repository includes:
   • A README.md that is effectively the main “front page” of the Whitepaper.
   • Additional Markdown files in the /pages directory (e.g., KEY_VALUE_STORE_SCHEMA.md, ACTOR_FILE.md).
   • Images stored in the /img folder.
   • Build scripts and a package.json (used to build a Table of Contents or check Markdown links).
   • Supporting metadata files (.gitignore, LICENSE.md).
4. The Astro site uses its own structure under /src/content/pages with lowercase filenames (e.g., key-value-store-schema.mdx).
5. We want to unify and automate the flow so that updates to @apify/actor-whitepaper automatically appear on the Astro website.
6. The Astro site is currently built and hosted by Vercel.

---

Requirements and goals

1. Single source of truth

The @apify/actor-whitepaper repository should contain all the canonical content. All new changes and additions happen here.

2. Automated syncing or processing

Changes in @apify/actor-whitepaper should feed into the Astro site automatically (e.g., via GitHub Actions), converting .md to .mdx as needed.

3. Filename normalization

Convert uppercase filenames in the Whitepaper repo to lowercase .mdx in the Astro site.

4. Image handling and optimization

Move images from /img in the Whitepaper repo to the appropriate location under src/content/pages/img in the Astro repo so that Sharp can properly handle them.

5. Formatting differences and MDX compatibility

Adjust syntax, insert frontmatter, and use Astro components (<Illustration>, <Picture>, <CodeSwitcher>) as needed.

6. Repository structure decision

Decide between keeping two separate repos (with automated sync) or merging into one. Consider commit history “noise” and potential reuse of Whitepaper content elsewhere.

7. Delivery method

Confirm whether to continue using Vercel or switch to GitHub Actions for build and deployment. Check build times and runner constraints.

---

Possible approaches

Two-repository model with automated sync

• Keep @apify/actor-whitepaper as a standalone repo:
  • All content changes happen here in Markdown.
  • Can remain public or internal.
• The Astro site repo remains separate:
  • Houses Astro configuration, styling, and additional site content.
  • Should be moved to the @apify account.
• Sync step with GitHub Actions:
  • On push/release to @apify/actor-whitepaper, an Action:
    • Clones/checks out the Astro site repo.
    • Copies/transforms .md to .mdx, normalizes filenames.
    • Copies images into the Astro repo.
    • Commits and pushes changes to the Astro site repo.
  • Astro site repo triggers its deployment pipeline (GitHub Actions or Vercel).

Pros

• Clear separation between content and site code.
• Keeps large doc changes out of the site commit history.

Cons

• Maintains two repos.
• More complexity in CI and potential debugging if sync fails.

Single monorepo (combining Whitepaper + Astro)

• Merge @apify/actor-whitepaper content into the Astro site’s codebase.
• Store raw content in /docs or a dedicated folder and import directly.
• Build and deploy from a single repository.

Pros

• One repository to manage, no sync pipeline.
• Simpler local development.

Cons

• Larger commit logs mixing docs and code.
• Harder to reuse Whitepaper content independently.

Subtree or submodule approach

• Keep @apify/actor-whitepaper as a submodule in the Astro site repository.
• A hybrid approach similar to the two-repo model, but possibly more complex for contributors.

Detailed considerations

Renaming & transforming files

• Rename KEY_VALUE_STORE_SCHEMA.md to key-value-store-schema.mdx etc.
• Insert frontmatter in MDX if required.
• Convert Markdown links, images, and code blocks to MDX-compatible syntax.

Formatting & linting

• The Astro site may run Prettier or custom linters on .mdx.
• Ensure formatting is consistent. Possibly run formatting after syncing.

MDX differences & components

• Astro uses custom MDX components like <Illustration>, <Picture>, <CodeExample>, <CodeSwitcher>.
• Standard Markdown images (![Alt](img.png)) might need converting into <Picture> or <Illustration> tags.
• Some .md files may contain comment placeholders like <!-- IMAGE: right, illuApifyStore, "Apify Actor Store" --> to guide the script on how to transform images.
• Code snippets might need grouping into <CodeSwitcher> components to provide tabbed code examples.

Special markup (e.g., layout reset)

• <div class="clear-both" /> might be needed after floated images to prevent wrapping.
• Insert these strategically where required, possibly hinted by placeholders in original .md files.

Frontmatter & layouts

• .mdx files may need a frontmatter block specifying a layout.
• The sync script can insert a standard frontmatter snippet at the top of each file.

Image handling

• Copy images from @apify/actor-whitepaper/img to astro-repo/src/content/pages/img/whitepaper (or similar).
• Ensure references are updated to the new paths, and if needed, wrap them in <Picture> or <Illustration>.

CI/CD

• Use GitHub Actions for syncing and possibly also for building/deploying.
• If Vercel continues to be used, just trigger a rebuild via a webhook after syncing.

Git history & scope

• The two-repo model keeps docs separate from the site code history.
• Single repo mixes everything. It might be okay for smaller projects, but less flexible.

Handling special site files (e.g., 404.mdx)

• The Astro site may have additional MDX files like 404.mdx or other custom pages not present in the Whitepaper repo.
• The sync script must not overwrite or remove these special site-level pages.
• Ensure the pipeline only updates files originating from the Whitepaper repository’s content, leaving the Astro site’s unique pages intact.

Recommendation & example workflow

Recommended: Two-repository model with automated sync

1. Keep @apify/actor-whitepaper as the source of truth.
2. Add a GitHub Action in @apify/actor-whitepaper that on push:
   • Checks out the Astro site repo.
   • Runs a custom script to:
     • Convert .md to .mdx, fix filenames, and insert frontmatter.
     • Transform images using placeholders into <Illustration> or <Picture>.
     • Group related code blocks into <CodeSwitcher> components.
     • Copy images to the correct folder.
     • Avoid overwriting special site pages (like 404.mdx).
   • Commits and pushes changes back to the Astro site repo.
3. The Astro site repo deploys automatically (via GitHub Actions or Vercel).

If we ever decide a single repo is simpler, we can merge them later. For now, this approach ensures clarity and maintainability.

Why not a single repo?

Keeping the Whitepaper separate is beneficial if it may be reused in multiple contexts (e.g., NPM packages and other sites). It reduces mixing deployment code with document content and keeps the content portable.

Another critical driver for keeping them separate is that the actor-whitepaper repository is intended as a narrowly focused RFC project, where contributors only see and discuss changes or issues related to the specification itself. If we unify the website build code in the same repo, people interested solely in the Whitepaper’s evolution would also receive GitHub notifications and issues about the site’s build or styling changes. This can be noisy and confusing for those who want to collaborate on the specs without going deep through the broader website infrastructure.

Potential pitfalls

• Sync script might need maintenance if the file structure changes drastically.
• Parsing comment placeholders and grouping code blocks may be non-trivial.
• Good documentation and stable workflows will reduce risks.

Conclusion

By preserving @apify/actor-whitepaper as the single source of truth and automatically updating the Astro site repository, we achieve the following:

• Automated syncing (no manual duplication).
• Clean separation of docs and site code.
• Potential for reusing the Whitepaper content elsewhere.

We can revisit this decision later if we prefer a single codebase. For now, the two-repo sync approach via GitHub Actions is the least disruptive path to unifying content and deployment.

/c @B4nan @mtrunkat @netmilk

[janbuchar]

TLDR, but how about putting the website sources in this (actor-whitepaper) repo and including the content directly?

[vancura]

Thanks for the suggestion, @janbuchar. Merging the website sources directly into the actor-whitepaper repo (this repo) and serving the content from here is an option. It would simplify the workflow since we’d have only one repository to maintain, and changes to the Whitepaper would be reflected immediately on the site.

However, I believe that keeping the two-repo structure would maintain a clean separation between the content and the site’s build/deployment logic. This approach preserves a standalone docs resource that can be reused or repurposed independently of the Astro site and keeps the commit history more organized for those focusing solely on content vs. site infrastructure.

That said, if the synchronization overhead proves too high in the future, we can always reconsider and merge everything into a single repository. This is just my perspective – my main concern is the complexity and noise in the Git history. However, I’m open to whatever solution makes the most sense for the team.

(Please see my explanation above in the section "Single monorepo (combining Whitepaper + Astro)")

[mtrunkat]

IMHO, the easiest way would be to place both in this repository and configure CI to publish the statically built https://whitepaper.actor site as GitHub pages. Our docs work this way, and the big advantage is that you can immediately preview all your changes on a locally run site. For example, you could edit markdown files and then run npm run to see what they will look like.

The disadvantage is that the whitepaper changelog would also include changes in the site, which is not great. I am not really opinionated on this topic, but if that's acceptable, then I'd rather go with this easier setup.

[netmilk]

The main reason for having two separate repos is that people interested in the actor-whitepaper repo won't care about the page github notifications/changes and issues. The intention is to have the actor-whitepaper repo as the "RFC" project as narrowly focused as possible without the necessity to navigate the structure of issue labels, a repo where people can collaborate, and get notified only about the specs, not all the implementation noise.

[jancurn]

+1 to the separation

[netmilk]

So @vancura, please add it to "Why not a single repo?" as a driver, please! :) I'd prefer not to edit your initial comment ;)

[vancura]

I added a new note under the “Why not a single repo?” section to explain the reasoning behind keeping the Whitepaper a focused RFC project separate from the site implementation. Thanks to @mtrunkat and @netmilk for pointing this out!

[vancura]

Below is a comprehensive plan to automatically transfer changes from the actor-whitepaper repository (the source of truth for all Whitepaper content – this very repo) into the actor-whitepaper-web Astro site repo, while ensuring that all changes land in a pull request (no direct merges to main).

Overview

We add a GitHub Action to the actor-whitepaper repo. Whenever someone pushes to the main branch of actor-whitepaper.

This workflow:

1. Checks out the latest content in actor-whitepaper.
2. Clones the actor-whitepaper-web repository (Astro site).
3. Runs a conversion script that:
   • Transforms .md → .mdx, injects Astro frontmatter, wraps images in <Picture> etc.
   • Copies images to the appropriate folders in actor-whitepaper-web.
4. Commits the resulting changes to a feature branch (e.g. sync/whitepaper-updates) in actor-whitepaper-web.
5. Creates or updates a PR in actor-whitepaper-web, so a human reviewer can merge the changes when ready.

This approach ensures no direct auto-merge to main: changes always go through a PR that must be explicitly approved and merged.

Notes

• Renovate bot will be part of this workflow.

Personal Access Token / secrets

We need a token with write access to actor-whitepaper-web.

• Minimal scopes: repo access is limited to that repository or a machine user with the same.
• This token is stored as a secret (e.g., WEB_REPO_WRITE_TOKEN) in actor-whitepaper.

The Action references that secret token for authenticated git push.

Notes

• I already asked @mtrunkat about the token setup.

Example workflow file

Below is a generic example of the YAML file (.github/workflows/sync-to-astro.yml) in the actor-whitepaper repo. This uses the GitHub CLI to create or update the pull request:

name: Sync Whitepaper to Astro (PR flow)

on:
  push:
    branches:
      - main

jobs:
  sync-content:
    runs-on: ubuntu-latest
    steps:
      # 1) Check out actor-whitepaper (the triggering repo)
      - name: Check out actor-whitepaper (source)
        uses: actions/checkout@v3

      # 2) Set up Node for the conversion script
      - name: Set up Node
        uses: actions/setup-node@v3
        with:
          node-version: 18

      # 3) Install dependencies for the conversion script
      - name: Install dependencies
        run: |
          npm install

      # 4) Clone actor-whitepaper-web (target)
      - name: Clone actor-whitepaper-web
        run: |
          git clone https://github.com/apify/actor-whitepaper-web.git

      # 5) Run the conversion
      - name: Convert and copy files
        run: |
          # For example:
          # node scripts/convert-md-to-mdx.js
          # This script would:
          #   - read .md files from actor-whitepaper
          #   - transform them to .mdx
          #   - copy them (and images) into actor-whitepaper-web
          # More details and the script contents are TBD; this is a general overview of what would happen here.
          echo "Conversion script done"

      # 6) Commit changes to a feature branch
      - name: Commit changes to a new branch
        run: |
          cd actor-whitepaper-web
          git config user.name "whitepaper-bot"
          git config user.email "actions@github.com"
          git checkout -b sync/whitepaper-updates || git checkout sync/whitepaper-updates

          if [ -n "$(git status --porcelain)" ]; then
            git add .
            git commit -m "chore: update from actor-whitepaper"
          else
            echo "No changes to commit."
          fi

      # 7) Push changes to that branch
      - name: Push changes
        if: always()
        run: |
          cd actor-whitepaper-web
          git remote set-url origin \
            https://x-access-token:${{ secrets.WEB_REPO_WRITE_TOKEN }}@github.com/apify/actor-whitepaper-web.git
          git push --set-upstream origin sync/whitepaper-updates

      # 8) Create or update pull request in actor-whitepaper-web
      - name: Install GitHub CLI
        uses: actions/checkout@v3
        with:
          repository: cli/cli
          path: gh-cli

      - name: Open or update PR
        run: |
          # Install GH CLI
          sudo make -C gh-cli all install

          cd actor-whitepaper-web
          # Check if a PR for this branch already exists
          if gh pr view sync/whitepaper-updates >/dev/null 2>&1; then
            echo "PR already exists, updating title/body..."
            gh pr edit sync/whitepaper-updates \
              --title "chore: update from actor-whitepaper" \
              --body "Automated sync from actor-whitepaper"
          else
            echo "Creating new PR..."
            gh pr create \
              --head sync/whitepaper-updates \
              --base main \
              --title "chore: update from actor-whitepaper" \
              --body "Automated sync from actor-whitepaper"

Notes

• The gh pr create step ensures we generate or update a pull request in actor-whitepaper-web.
• If the PR already exists, we update it instead of creating duplicates.
• Reviewers can inspect and approve the PR. Only after that manual approval does it merge to main (NB, a Renovate comment will be included in the PR).

Benefits & details

1. Manual approval: We maintain a PR-based workflow by pushing changes to a feature branch. No code merges to main automatically.
2. Security:
   • Use a minimal-scope PAT or machine user token with write permission to actor-whitepaper-web only.
   • Store it in the actor-whitepaper repo as a secret.
   • Keep the logs from revealing the token (GitHub masks ${{ secrets.* }} by default).
3. No infinite loops: because the push is only from actor-whitepaper → actor-whitepaper-web, and we do not have a reverse sync, we avoid the scenario of repeated triggers.
4. Adaptability: we can use @peter-evans/create-pull-request or other dedicated actions instead of the GitHub CLI.

---

This solution gives us an automated pipeline where updates to actor-whitepaper create or update a PR in actor-whitepaper-web, requiring a human reviewer to merge. It enforces safe review processes, ensures transformations from .md to .mdx are handled automatically, and preserves the final decision to merge in a manual step.

Note: The conversion script is TBD.

Thoughts?

[vancura]

@seyhello helped me with the solution.

We haven't done anything like this anywhere yet, but I tried it. It won't work without a token, but we are already prepared for it at the organizational level, and the token for the service account is already generated there, so you just need to use it. This is how it worked for me:

jobs:
  sync:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - name: Checkout Repo A
        uses: actions/checkout@v4

      - name: Checkout Repo B
        uses: actions/checkout@v4
        with:
          repository: apify/repo
          token: ${{ secrets.TOKEN_NAME }}

      - name: Make Changes in Repo B
        run: |
          git checkout -b release/new-branch
          echo "Update from Repo A" >> update.txt
          git config --global user.email "user@test.com"
          git config --global user.name "Test User"
          git add update.txt
          git commit -m "Automated update from Repo A"

      - name: Push Changes
        run: |
          git push origin release/new-branch

Regarding creating a PR in the second rep, we have a ready-made action and workflow, so you could already use it. Basically, in that destination rep, you made short workflows of this type:

# This workflow create release pull request with changelog in description
name: manage release

on:
  push:
    branches:
      - release/**

  workflow_dispatch:

jobs:
  manage_release:
    uses: apify/workflows/.github/workflows/open_pull_request.yaml@v0.22.0
    with:
      changelogScopes: '{ "Worker": ["worker", "app"] }'
    secrets:
      githubToken: ${{ secrets.TOKEN_NAME }}

Thank you, @seyhello!