deploy: check for broken links

Broken links are quite an annoyance for readers, and at least for links
that point to pages within the same static website, there are tools to
help identify those.

One such tool is called `lychee`. It has already a number of very useful
options and even sports a GitHub Action for easy integration into GitHub
workflows.

To accommodate the migration of https://git-scm.com/ from a Rails App to
a static site generated using Hugo, lychee was taught the trick needed
to support checking links in a static website with "pretty URLs" (i.e.
URLs lacking the `.html` file extension even though the files backing
those URLs do have that extension, something GitHub Pages supports).

With this mode, the automation that deploys https://git-scm.com/ can
make use of that link checker.

Seeing as broken links often originate from repositories outside of
https://github.com/git/git-scm.com's control, rather than failing
deployment when broken links are detected, let's follow the "best
effort" strategy and open a ticket about the broken links while still
letting the deployments complete.

In PR builds, links are also checked, and broken links will let them
fail.

While it was tempting to use a convenient GitHub Action like
`peter-evans/create-issue-from-file` to open the GitHub issue, there is
no readily-available Action to either open, update, or update & close
such a ticket, depending on the outcome of the link check and whether
such a ticket exists already and is open. Therefore, I opted for a more
verbose (and much more powerful) `actions/github-script` step to perform
this part of the job.

Based-on-feedback-by: Todd Zullinger <[email protected]>
Helped-by: Matthias Endler <[email protected]>
Helped-by: Thomas Zahner <[email protected]>
Signed-off-by: Johannes Schindelin <[email protected]>

Author: dscho

.github/actions/deploy-to-github-pages/action.yml

@@ -9,6 +9,11 @@ description: 'Runs Hugo and Pagefind and then deploys the result to GitHub Pages
 # environment:
 #   name: github-pages
 #   url: ${{ steps.<id-of-deployment-step>.outputs.url }}
+inputs:
+  github-token:
+    description: The GitHub token used to create an authenticated client
+    default: ${{ github.token }}
+    required: true
 outputs:
   url:
     description: The URL to which the site was deployed
@@ -62,3 +67,63 @@ runs:
     - name: deploy
       id: deploy
       uses: actions/deploy-pages@v4
+
+    - name: construct `--remap` option for lychee
+      id: remap
+      shell: bash
+      run: |
+        base_url='${{ steps.pages.outputs.base_url }}'
+        echo "result=$(echo "$base_url" |
+          sed 's|^\(.*\)\(/git-scm\.com\)$|(\1)?\2(.*)|') file://$PWD/public\$2" \
+          >>$GITHUB_OUTPUT
+
+    - name: check for broken links
+      id: lychee
+      uses: lycheeverse/lychee-action@d4128702eae98bbc5ecf74df0165a8156c80920a # until an official version is out that includes https://github.com/lycheeverse/lychee/pull/1422
+      with:
+        lycheeVersion: nightly # until an official version includes https://github.com/lycheeverse/lychee/pull/1422
+        args: >-
+          --offline
+          --fallback-extensions html
+          --base '${{ steps.pages.outputs.base_url }}'
+          --remap '${{ steps.remap.outputs.result }}'
+          --exclude file:///path/to/repo.git/
+          --exclude file:///caminho/para/o/reposit%C3%B3rio.git/
+          --exclude file:///ruta/a/repositorio.git/
+          --exclude file:///sl%C3%B3%C3%B0/til/hirsla.git/
+          --exclude file:///Pfad/zum/Repo.git/
+          --exclude file:///chemin/du/d%C3%A9p%C3%B4t.git/
+          --exclude file:///srv/git/project.git
+          public/
+        output: lychee.md
+        jobSummary: true
+        fail: false
+        failIfEmpty: false # needed because its default overrides `fail = false`
+
+    - name: ${{ env.lychee_exit_code != '0' && 'maybe close' || 'open or update' }} link checker issue
+      uses: actions/github-script@v7
+      with:
+        github-token: ${{ inputs.github-token }}
+        script: |
+          const fs = await import('fs')
+          // GitHub issues can only have 64k characters in their body
+          const body = (s => {
+            if (s.length < 65535) return s
+            return s.replace(/^([^]{0,65000}\n)[^]*\n(.+)\n?$/, '$1\n[...]\n\n$2')
+          })(await fs.promises.readFile('lychee.md', 'utf8'))
+
+          const req = { owner: context.repo.owner, repo: context.repo.repo }
+          const q = `"Link+Checker+Report"+in:title+is:issue+label:linkchecker+is:open+repo:${req.owner}/${req.repo}`
+          const issues = await github.rest.search.issuesAndPullRequests({ ...req, q, sort: 'created', per_page: 1 })
+
+          if (issues.data.items.length === 0) {
+            if (process.env.lychee_exit_code !== '0') {
+              await github.rest.issues.create({ ...req, title: 'Link Checker Report', body, labels: ['linkchecker'] })
+            }
+          } else {
+            req.issue_number = issues.data.items[0].number
+            await github.rest.issues.createComment({ ...req, body })
+            if (process.env.lychee_exit_code === '0') {
+              await github.rest.issues.update({ ...req, state: 'closed' })
+            }
+          }

.github/workflows/ci.yml

@@ -57,3 +57,22 @@ jobs:
           }
         done
         exit $res
+
+    - name: check for broken links
+      id: lychee
+      uses: lycheeverse/lychee-action@d4128702eae98bbc5ecf74df0165a8156c80920a # until an official version is out that includes https://github.com/lycheeverse/lychee/pull/1422
+      with:
+        lycheeVersion: nightly # until an official version includes https://github.com/lycheeverse/lychee/pull/1422
+        args: >-
+          --offline
+          --fallback-extensions html
+          --exclude file:///path/to/repo.git/
+          --exclude file:///caminho/para/o/reposit%C3%B3rio.git/
+          --exclude file:///ruta/a/repositorio.git/
+          --exclude file:///sl%C3%B3%C3%B0/til/hirsla.git/
+          --exclude file:///Pfad/zum/Repo.git/
+          --exclude file:///chemin/du/d%C3%A9p%C3%B4t.git/
+          --exclude file:///srv/git/project.git
+          public/
+        output: lychee.md
+        jobSummary: true

.github/workflows/deploy.yml

@@ -12,6 +12,7 @@ jobs:
     permissions:
       pages: write # to deploy to GitHub Pages
       id-token: write # to verify that the deployment source is legit
+      issues: write # to open tickets upon broken links
     environment:
       name: github-pages
       url: ${{ steps.deploy.outputs.url }}

.github/workflows/update-book.yml

@@ -109,6 +109,7 @@ jobs:
       contents: write # to push changes (if any)
       pages: write # to deploy to GitHub Pages
       id-token: write # to verify that the deployment source is legit
+      issues: write # to open tickets upon broken links
     environment:
       name: github-pages
       url: ${{ steps.deploy.outputs.url }}

.github/workflows/update-download-data.yml

@@ -13,6 +13,7 @@ jobs:
       contents: write # to push changes (if any)
       pages: write # to deploy to GitHub Pages
       id-token: write # to verify that the deployment source is legit
+      issues: write # to open tickets upon broken links
     environment:
       name: github-pages
       url: ${{ steps.deploy.outputs.url }}

.github/workflows/update-git-version-and-manual-pages.yml

@@ -18,6 +18,7 @@ jobs:
       contents: write # to push changes (if any)
       pages: write # to deploy to GitHub Pages
       id-token: write # to verify that the deployment source is legit
+      issues: write # to open tickets upon broken links
     environment:
       name: github-pages
       url: ${{ steps.deploy.outputs.url }}

.github/workflows/update-translated-manual-pages.yml

@@ -37,6 +37,7 @@ jobs:
       contents: write # to push changes (if any)
       pages: write # to deploy to GitHub Pages
       id-token: write # to verify that the deployment source is legit
+      issues: write # to open tickets upon broken links
     environment:
       name: github-pages
       url: ${{ steps.deploy.outputs.url }}

README.md

@@ -160,6 +160,10 @@ The [list of GUI clients](https://git-scm.com/downloads/guis) has been construct
 
 * https://pagefind.app/
 
+### Lychee (link checker)
+
+* https://lychee.cli.rs/
+
 ## License
 
 The source code for the site is licensed under the MIT license, which you can find in