Use npm 8 in CI (to allow overriding dep versions)

This requires some fixes, including:
- Updates to support the new base image, which otherwise has some issues
  with npm permissions in this case.
- Tweaks to the release build process to ensure that the prepare script
  keeps working correctly with the different lifecycle behaviour.
- Pinning prebuild-install to v7, because v5 (used by some deps) doesn't
  support the env vars for cross-platform builds with npm v7+.

Regarding the base image: previously, the image had a default root
user, so the checkout was made as root, but GHA always runs run steps
as user 1001. After updating npm, it started to fail to install due to
permissions issues, because it was running as user 1001 while the
checkout was made as root.

The base image has now been updated to default to a 1001 user called
'build-user' who has passwordless sudo & docker permissions. With a
few small changes here, we can support that and everything should work
correctly with a single consistent user used in every step.

Author: pimterry

.github/workflows/ci.yml

@@ -7,7 +7,7 @@ jobs:
     name: Build & test
     runs-on: ubuntu-18.04
     container:
-      image: httptoolkit/act-build-base
+      image: httptoolkit/act-build-base:v2.0.0
       options: "--privileged" # Required for DIND tests
     env:
       DIND_HOST: unix:///tmp/docker.sock
@@ -18,6 +18,8 @@ jobs:
         with:
           node-version: 14.18.3
 
+      - run: npm install -g npm@^8
+
       - name: Set up JDK 11 for Java testing
         uses: actions/setup-java@v1
         with:
@@ -31,20 +33,20 @@ jobs:
 
       - name: Prepare test environment
         run: |
-          apt-get update
-          apt-get install p7zip-full xterm
+          sudo apt-get update
+          sudo apt-get install p7zip-full xterm
 
           wget https://downloads.slack-edge.com/linux_releases/slack-desktop-4.12.2-amd64.deb
-          apt install ./slack-desktop-4.12.2-amd64.deb
+          sudo apt install ./slack-desktop-4.12.2-amd64.deb
 
-          cp test/no-sandbox-docker-wrapper.sh /usr/local/bin/google-chrome
-          cp test/no-sandbox-docker-wrapper.sh /usr/local/bin/slack
+          sudo cp test/no-sandbox-docker-wrapper.sh /usr/local/bin/google-chrome
+          sudo cp test/no-sandbox-docker-wrapper.sh /usr/local/bin/slack
 
           # Fix $HOME perms to make Firefox happy
-          chown $(whoami) $HOME
+          sudo chown $(whoami) $HOME
 
           # Start a DIND docker host, running in the background
-          dockerd -H $DIND_HOST &
+          sudo dockerd -H $DIND_HOST &
           sleep 5
 
           # Pre-pull lots of the Docker images we'll want to use later

pack.ts

@@ -20,6 +20,15 @@ const spawn = (command: string, args: string[] = [], options: SpawnOptions = {})
     });
 }
 
+/**
+ * The process here is that we create a clone of the whole codebase, but keeping the latest
+ * build output, and make a couple of small modifications to manage the build process, and then
+ * we handle off to Oclif to rebuild once for each target operating system.
+ *
+ * Oclif will run "npm pack", then "npm unpack" into a temp directory, "npm install" there, inject
+ * the appropriate scripts & node binaries to make everything work nicely, and then bundled us up
+ * a nice tarball ready for deployment.
+ */
 const packageApp = async () => {
     console.log('Preparing packaging directory');
     await fs.emptyDir(OUTPUT_DIR);
@@ -59,7 +68,12 @@ const packageApp = async () => {
         .keyBy(_.identity)
         .mapValues((pkg: string) => pLockJson.dependencies[pkg].version);
 
-    delete pJson.scripts.prepack; // We don't want to rebuild
+    // Oclif is going to re-run install, and there's a couple of extra files that will be required to make
+    // that work, which aren't normally included by the "npm pack"/"npm unpack" flow, so we manually pull
+    // them across here:
+    pJson.scripts.preinstall = `cp ../../prepare.ts . && cp ../../overrides/js/package-lock.json overrides/js`;
+
+    delete pJson.scripts.prepack; // We don't want to rebuild - all built code will be in the packed content
     await fs.writeJson(path.join(OUTPUT_DIR, 'package.json'), pJson);
 
     const buildScript = path.join(OUTPUT_DIR, 'build-release.sh');