Merge branch 'main' into docs/cname-resources

Author: mcanouil

.github/workflows/README.md

@@ -0,0 +1,54 @@
+# About our workflows
+
+This repository uses different workflows to automatically updates the published websites
+
+- <https://quarto.org/> - The documentation for Quarto, published from the `main` branch
+- <https://prerelease.quarto.org/> - The documentation for Quarto prerelease, published from the `prerelease` branch
+
+It also configures some website preview for pull requests, which are published to Netlify as deploy previews. The use non-production deploy and can be consulted to `deploy-preview-<PR number>.quarto.org`. 
+
+Note that technically, <prerelease.quarto.org> is also a deploy preview on Netlify (non-production website) but with a fixed URL.
+
+## Workflow files: What they do
+
+- `publish.yml` - The workflow that publishes the documentation from `main` and `prerelease` branches to netlify. 
+  - For stable release version, the doc is deployed from `main`, and published using `quarto publish netlify` command to production website on Netlify.
+  - For prerelease version, the doc is deployed from `prerelease`, and published using Github Action `nwtgck/actions-netlify` which use Netlify API to deploy to a non-production website on Netlify.
+    The prerelease website is build using a specific profile at render time, set in action using `QUARTO_PROFILE` environment variable.
+
+- `preview.yml` - The workflow that creates a deploy preview for pull requests. It uses the same action `nwtgck/actions-netlify` to deploy to a non-production website on Netlify. The URL is `deploy-preview-<PR number>.quarto.org`.
+  - PR previews are configured for any PR to `main` or`prerelease` branches.
+  - They are automically created and updated when the PR is created and updated by a user with Contributor role. 
+  - For external PR, the preview can be triggered by adding a comment `/deploy-preview` on the PR.
+  - Any rendering to prerelease also uses a specific profile at render time, set in action using `QUARTO_PROFILE` environment variable.
+
+- `update-downloads.yml` - This workflow is triggered by a cron schedule. It retrieves information about latest release and prerelease on `quarto-dev/quarto-cli` repository and updates the download links on the website.
+  - If there is a new version detected, it will commit the modified files and trigger a deploy of the website calling `publish.yml` workflow with `workflow_call` event trigger.
+  - This applies to both `main` and `prerelease` branches
+    - For `main` branch, we use `stefanzweifel/git-auto-commit-action` to detect changes and commit them to the `main` branch
+    - For `prerelease` branch, we use `git cherry-pick` the commit from main following previous above step. 
+    - Then we trigger the `publish.yml` workflow with `workflow_call` event trigger for each of the branch.
+
+- `upload-index.yml` - This workflow is triggered by a cron schedule. It updates the indexes for Algolia search engine, which powers the sites search. 
+  - `search.json` is built when the website is rendered and then it is deployed to the website.
+  - This index file is retrieved on deployed website to be updated on Algolia.
+  - Both `quarto.org` and `prerelease.quarto.org` indexes are updated in the same run - they each use one specific algolia index
+
+- `port-to-prerelease.yml` - This workflow is used to sync changes made to main for quarto.org to prerelease branch for prerelease.quarto.org. 
+  - It is triggered when a PR is merged in to `main`. It can also be triggered manually by adding a comment `/sync-prerelease` on a merged PR.
+  - This workflow uses [`korthout/backport-action`](https://github.com/korthout/backport-action) to create a PR with the changes merged into `main` branch to be synced to `prerelease` branch.
+  - It will also write a new `/deploy-preview` comment in the new PR to trigger the preview deployment from `preview.yml`.
+  - This is possible because it uses a fine-grained PAT token which allows a workflow to trigger another using usual event (GITHUB_TOKEN does not allow that usually). This is configured in repo secrets.
+
+## Netlify Configurations
+
+- This repo has a `_redirects` file in the root directory. Otherwise, configuration are made in NETLIFY UI. Quarto website is inside Posit Netlify account. 
+- Automatic builds are turned off for this repo on Netlify because we currently need to render with Quarto CLI in Gihub Action CI before publishing the results. 
+  - API deploy is used currently, using npm netlify package through `nwtgck/actions-netlify` github actions. One think to know: Netlify has removed `deploy()` function from its npm package in version 7 and recommends now to use Netlify CLI instead. But we are still using the package for now in CI, but also for `quarto publish netlify`.
+  - Deploy previews are possible with manual deploy using an alias configuration for the deploy command; This allows us to `deploy-preview-<PR number>` and `prerelease` alias to be used for the deploy previews. 
+  - This means our netlify website don't use automatic builds and automatic Branch deploy - Only deploy previews. 
+  - The website uses [Automatic Deploy Subdomains](https://docs.netlify.com/domains-https/custom-domains/automatic-deploy-subdomains/#use-a-new-custom-domain-for-your-automatic-deploy-subdomain) to have `prerelease.quarto.org` or `deploy-preview-<PR number>.quarto.org` working.
+
+## Custom Domain
+
+- `quarto.org` is registered in external registrar, but associated with Netlify DNS allowing use to use [custom domains](https://docs.netlify.com/domains-https/custom-domains/multiple-domains/).

.github/workflows/actions/release-info/action.yml

@@ -27,5 +27,5 @@ inputs:
     required: true
 
 runs:
-  using: "node16"
+  using: "node20"
   main: "index.js"

.github/workflows/port-to-prerelease.yml

@@ -0,0 +1,65 @@
+name: Port change to prerelease
+on:
+  pull_request_target:
+    types: [closed]
+  issue_comment:
+    types: [created]
+
+permissions:
+  contents: write # so it can comment
+  pull-requests: write # so it can create pull requests
+  checks: write # so it can create a check run
+
+jobs:
+  sync-to-prerelease:
+    name: Backport pull request
+    runs-on: ubuntu-latest
+    # Don't run on closed unmerged pull requests and only PRs to main
+    # or on comment made by Github Action Bot
+    if: > 
+      (
+        github.event.pull_request.merged && 
+        github.event.pull_request.base.ref == 'main'
+      ) || (
+        github.event_name == 'issue_comment' &&
+        github.event.issue.pull_request &&
+        github.event.comment.user.id != 41898282 &&
+        startsWith(github.event.comment.body, '/sync-prerelease')
+      )
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Create backport pull requests
+        id: synced-pr
+        # https://github.com/korthout/backport-action/releases/tag/v3.0.0
+        uses: korthout/backport-action@7e7cb1977d64d20339e8fdec12c0a1a065371de7
+        with:
+          cherry_picking: auto
+          branch_name: sync-${pull_number}-to-${target_branch}
+          merge_commits: skip
+          pull_title: '[Sync to ${target_branch}] ${pull_title}'
+          pull_description: |
+            # Description
+            Sync of #${pull_number} to `${target_branch}`.
+
+            ## Original PR
+
+            ### ${pull_title}
+            ${pull_description}
+          # Don't trigger on labels, and always PR to prerelease
+          label_pattern: ''
+          target_branches: prerelease
+
+      - name: Trigger deploy using a comment
+        if: steps.synced-pr.outputs.was_successful == 'true'
+        uses: actions/github-script@v7
+        with:
+          # special fine-grained token for issue comment trigger
+          github-token: ${{ secrets.ISSUE_COMMENT_TRIGGER_PAT }}
+          script: |
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: "${{ steps.synced-pr.outputs.created_pull_numbers }}",
+              body: "/deploy-preview",
+            });

.github/workflows/preview.yml

@@ -1,11 +1,18 @@
 on:
   pull_request:
-    branches: [main, v1.4]
+    branches: [main, prerelease]
   issue_comment:
     types: [created]
 
 name: Deploy Preview
 
+concurrency:
+  # Use github.event.pull_request.number on pull requests, so it's unique per pull request
+  # Use github.event.issue.number on issue comments, so it's unique per comment
+  # Use github.ref on other branches, so it's unique per branch
+  group: ${{ github.workflow }}-${{ (github.event.pull_request && format('PR-{0}', github.event.pull_request.number)) || ( github.event.issue && format('comment-{0}', github.event.issue.number) ) || github.ref }}
+  cancel-in-progress: true
+
 jobs:
   is-external-pr:
     # Be helpful with reviewer and remind them to trigger a deploy preview if the PR is from a fork.
@@ -17,13 +24,41 @@ jobs:
           echo "::error title=Manual action required for preview::PR from fork can't be deployed as preview to Netlify automatically. Use '/deploy-preview' command in comments to trigger the preview manually."
         shell: bash
 
+  role-of-commenter:
+    if: github.event.issue.pull_request
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check if commenter is a member, owner or collaborator
+        id: commenter-check
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          commenter_role=$(gh api repos/$GITHUB_REPOSITORY/collaborators/${{ github.event.comment.user.login }}/permission --jq '.permission')
+          echo "commenter_role=$commenter_role" >> "$GITHUB_OUTPUT"
+          echo "author_association=${{ github.event.comment.author_association }}" >> "$GITHUB_OUTPUT"
+        shell: bash
+
   build-deploy-preview:
     # Deploy a preview only if 
     # - the PR is not from a fork,
-    # - requested by PR comment /deploy-preview
-    if: |
-      (github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork != true) || 
-      (github.event.issue.pull_request && (github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'OWNER' || github.event.comment.author_association == 'COLLABORATOR') && startsWith(github.event.comment.body, '/deploy-preview')) 
+    # - requested by PR comment /deploy-preview, from a repo user or github action bot (user id 41898282)
+    # FIXME: We need to change the way we filter because somehow some MEMBER in API are seen as CONTRIBUTOR in CI
+    if: >
+      (
+        github.event_name == 'pull_request' && 
+        github.event.pull_request.head.repo.fork != true
+      ) || 
+      (
+        github.event.issue.pull_request && 
+        (
+          github.event.comment.user.id == '41898282' ||
+          github.event.comment.user.login == 'gordonwoodhull' ||
+          github.event.comment.author_association == 'MEMBER' || 
+          github.event.comment.author_association == 'OWNER' || 
+          github.event.comment.author_association == 'COLLABORATOR'
+        ) && 
+        startsWith(github.event.comment.body, '/deploy-preview')
+      ) 
       
     runs-on: ubuntu-latest
     steps:
@@ -44,12 +79,35 @@ jobs:
         with:
           version: ${{ steps.github-release.outputs.version }}
 
+      - name: Is it for prerelease website ?
+        id: prerelease-docs-check
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          if [ "${{ github.event.pull_request.base.ref }}" == "prerelease" ]; then
+            echo "is_prerelease_docs=true" >> "$GITHUB_OUTPUT"
+          elif [ -n "${{ github.event.issue.pull_request.url }}" ]; then
+            # This will trigger when not pull request event, so a PR comment event. 
+            # we need to get the base info from PR number the comment is made in
+            base_ref=$(gh -R $GITHUB_REPOSITORY pr view ${{ github.event.issue.number }} --json baseRefName --jq '.baseRefName')
+            if [ "$base_ref" == "prerelease" ]; then
+              echo "is_prerelease_docs=true" >> "$GITHUB_OUTPUT"
+            else
+              echo "is_prerelease_docs=false" >> "$GITHUB_OUTPUT"
+            fi
+          else
+            echo "is_prerelease_docs=false" >> "$GITHUB_OUTPUT"
+          fi
+        shell: bash
+
       - name: Render 
         uses: quarto-dev/quarto-actions/render@v2
+        env:
+          QUARTO_PROFILE: ${{ steps.prerelease-docs-check.outputs.is_prerelease_docs == 'true' && 'prerelease-docs' || '' }}
 
       - name: Deploy Preview to Netlify as preview
         id: netlify-deploy
-        uses: nwtgck/actions-netlify@v2
+        uses: nwtgck/actions-netlify@v3
         env:
           NETLIFY_SITE_ID: 2a3da659-672b-4e5b-8785-e10ebf79a962
           NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}

.github/workflows/publish.yml

@@ -1,24 +1,35 @@
 on:
   workflow_dispatch:
   push:
-    branches: main
+    branches: 
+      - main
+      - prerelease
   workflow_call:
     inputs:
       ref: 
         required: true
         description: 'The ref to trigger the workflow on'
         type: string
+      prerelease:
+        required: true
+        description: 'Whether to deploy prerelease or release site'
+        type: boolean
+        default: false
 
 name: Quarto Publish
 
+concurrency:
+  group: ${{ format('{0} - {1} - {2}', github.workflow, github.ref, inputs.prerelease && 'prerelease' || 'main') }}
+  cancel-in-progress: true
+
 jobs:
   build-deploy:
     runs-on: ubuntu-latest
     steps:
       - name: Check out repository
         uses: actions/checkout@v4
         with:
-          # if called fomr workflow_call event, checkout the ref otherwise use default
+          # if called from workflow_call event, checkout the ref otherwise use default
           ref: ${{ inputs.ref || '' }}
 
       - name: Get latest pre-release from github
@@ -33,8 +44,48 @@ jobs:
         with:
           version: ${{ steps.github-release.outputs.version }}
 
-      - name: Render and Publish 
+      - name: Render 
+        uses: quarto-dev/quarto-actions/render@v2
+        env:
+          QUARTO_PROFILE: ${{ (inputs.prerelease || (contains(fromJSON('["push", "workflow_dispatch"]'), github.event_name) && github.ref == 'refs/heads/prerelease')) && 'prerelease-docs' || '' }}
+
+      - name: Publish release website
+        # Only do this step if 
+        # - workflow is called for release deploy, with inputs.prerelease being false. It should not trigger if inputs.prerelease is null.
+        #   and in GHA null == false is true. That is why we convert to string and compare.
+        # - workflow is triggered by push event on main branch
+        # - workflow is triggered by workflow_dispatch event 
+        #   (we need to protect also from workflow_dispatch inherited from workflow_call event - where inputs.prerelease is defined)
+        if: ${{ format('{0}', inputs.prerelease) == 'false' || (!inputs.prerelease && contains(fromJSON('["push", "workflow_dispatch"]'), github.event_name) && github.ref == 'refs/heads/main') }}
         uses: quarto-dev/quarto-actions/publish@v2
         with:
           target: netlify
           NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
+          render: false
+
+      - name: Deploy Prerelease website to Netlify
+        # Only do this step if 
+        # - workflow is called for release deploy with inputs.prerelease being true. It should not trigger if inputs.prerelease is null.
+        #   Though as in GHA null means false, no need to cast to string
+        # - workflow is triggered by push event on main branch
+        # - workflow is triggered by workflow_dispatch event 
+        #   (we need to protect also from workflow_dispatch inherited from workflow_call event - where inputs.prerelease is defined)
+        if: ${{ inputs.prerelease || (format('{0}', inputs.prerelease) != 'false' && contains(fromJSON('["push", "workflow_dispatch"]'), github.event_name) && github.ref == 'refs/heads/prerelease') }}
+        id: netlify-deploy
+        uses: nwtgck/actions-netlify@v3
+        env:
+          NETLIFY_SITE_ID: 2a3da659-672b-4e5b-8785-e10ebf79a962
+          NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
+        with:
+          publish-dir: './_site'
+          production-deploy: false
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          deploy-message: |
+            Deploy from GHA: ${{ format('manual from {0} at commit {1}', github.ref_name, github.sha) }}
+          alias: prerelease
+          # these all default to 'true'
+          enable-pull-request-comment: false
+          enable-commit-comment: false
+          enable-commit-status: true
+          overwrites-pull-request-comment: false
+        timeout-minutes: 1

.github/workflows/update-downloads.yml

@@ -10,11 +10,13 @@ jobs:
     outputs:
       changes_detected: ${{ steps.auto-commit.outputs.changes_detected }}
       commit: ${{ steps.auto-commit.outputs.commit_hash }}
+      commit_prerelease: ${{ steps.cherry-pick-prerelease.outputs.commit_hash }}
     steps:
       - name: Checkout Repo
         uses: actions/checkout@v4
 #        with:
 #          token: ${{ secrets.COMMIT_PAT }}
+
       - name: Regenerate Download Json
         uses: ./.github/workflows/actions/release-info
         with:
@@ -24,8 +26,9 @@ jobs:
           redirects-path: '_redirects'
           redirects-template: /download/latest/$$prefix$$-$$suffix$$.$$extension$$
           pre-redirects-template: /download/prerelease/$$prefix$$-$$suffix$$.$$extension$$
-          github-token: ${{ github.token }}          
-      - name: Commit Changes
+          github-token: ${{ github.token }}  
+
+      - name: Commit Changes to main branch
         id: auto-commit
         uses: stefanzweifel/git-auto-commit-action@v5
         with:
@@ -34,14 +37,39 @@ jobs:
           commit_user_email: [email protected]
           commit_author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
           skip_checkout: true
+        
+      - name: Cherry-pick change to prerelease branch
+        id: cherry-pick-prerelease
+        if: ${{ steps.auto-commit.outputs.changes_detected == 'true' }}
+        run: |
+          git config --global user.name 'github-actions[bot]'
+          git config --global user.email '41898282+github-actions[bot]@users.noreply.github.com'
+          git checkout prerelease
+          git cherry-pick ${{ steps.auto-commit.outputs.commit_hash }}
+          git push origin prerelease
+          # set an output for github action with the resulted commit of cherry-pick command
+          echo "commit_hash=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
 
   # If a new commit has been made with updated downloads, 
   # then publish the changes using the new commit as ref to checkout
   # in the reusable publish workflow
-  publish-changes:
+  publish-changes-release:
     needs: [ update-downloads ]
     if: ${{ needs.update-downloads.outputs.changes_detected == 'true' }}
     uses: ./.github/workflows/publish.yml
     with:
       ref: ${{ needs.update-downloads.outputs.commit }}
+      prerelease: false
+    secrets: inherit
+
+  # If a new commit has been made with updated downloads, 
+  # then publish the changes using the new commit as ref to checkout
+  # in the reusable publish workflow
+  publish-changes-prerelease:
+    needs: [ update-downloads ]
+    if: ${{ needs.update-downloads.outputs.changes_detected == 'true' }}
+    uses: ./.github/workflows/publish.yml
+    with:
+      ref: ${{ needs.update-downloads.outputs.commit_prerelease }}
+      prerelease: true
     secrets: inherit