Merge branch 'master' into drop-make

Author: alexreinking

README.md

@@ -8,34 +8,13 @@ $ uv sync --all-packages
 
 # Master configuration
 
-## Web server settings
-
-Using your production-quality web server of choice (Apache, Nginx, etc.), choose
-a URL at which to host the master. Call this `BUILDBOT_WWW`. Then, set up a
-reverse proxy for the buildbot webserver (on port 8012). For Apache, your
-configuration might look like:
-
-```
-ProxyPass /ws ws://localhost:8012/ws
-ProxyPassReverse /ws ws://localhost:8012/ws
-ProxyPass / http://localhost:8012/
-ProxyPassReverse / http://localhost:8012/
-
-SetEnvIf X-Url-Scheme https HTTPS=1
-ProxyPreserveHost On
-```
-
-Note that you will need to enable `proxy_wstunnel` for this to work (via
-`a2enmod`). It is essential that HTTPS only is used (for security).
-
-**Close port 8012 to the internet.** If you can't have port 9990 open, redirect
-another port to it. Whichever port this is, call it `MASTER_PORT`.
-
-Make a note of your master's IP address. Call this `MASTER_ADDR`.
+The master is deployed via [Docker Compose][dc], which manages three services:
+a PostgreSQL database, the Buildbot master, and a [Caddy] reverse proxy with
+automatic HTTPS.
 
 ## Secrets
 
-Four secrets control authentication with external users and servers. These will
+Five secrets control authentication with external users and servers. These will
 need to be determined before starting up a new master.
 
 1. Obtain a [GitHub personal access token](https://github.com/settings/tokens)
@@ -48,16 +27,30 @@ need to be determined before starting up a new master.
    Call this `WEBHOOK_SECRET`.
 4. Choose a password for the `halidenightly` user to authenticate with the web
    interface. Call this `WWW_PASSWORD`.
+5. Generate a password for the PostgreSQL database. Call this `DB_PASSWORD`.
+   This is only needed when using PostgreSQL (i.e., when
+   `HALIDE_BB_MASTER_DB_URL` contains `{DB_PASSWORD}`). The default SQLite
+   backend does not require it.
 
 A convenient command for generating a secure secret is `openssl rand -hex 20`.
 
-## GitHub Configuration
+Write all the secrets to the corresponding files:
+
+```console
+$ echo "$GITHUB_TOKEN" > secrets/github_token.txt
+$ echo "$WORKER_SECRET" > secrets/halide_bb_pass.txt
+$ echo "$WEBHOOK_SECRET" > secrets/webhook_token.txt
+$ echo "$WWW_PASSWORD" > secrets/buildbot_www_pass.txt
+$ echo "$DB_PASSWORD" > secrets/db_password.txt
+```
+
+## GitHub configuration
 
 Make your way to the Webhooks section of your repository settings. The url is
 `https://github.com/{owner}/{repo}/settings/hooks`. The following settings are
 the correct ones:
 
-1. **Payload URL:** `$BUILDBOT_WWW/change_hook/github`
+1. **Payload URL:** `https://buildbot.halide-lang.org/master/change_hook/github`
 2. **Content type:** `application/json`
 3. **Secret:** `$WEBHOOK_SECRET`
 4. **SSL verification:** Select _Enable SSL verification_
@@ -67,47 +60,91 @@ the correct ones:
 
 ## Starting the master
 
-First, write all the secrets to the corresponding files:
+Choose a directory to hold artifacts from package builds (the default is
+`./data/artifacts`):
 
 ```console
-$ echo "$GITHUB_TOKEN" > secrets/github_token.txt
-$ echo "$WORKER_SECRET" > secrets/halide_bb_pass.txt
-$ echo "$WEBHOOK_SECRET" > secrets/webhook_token.txt
-$ echo "$WWW_PASSWORD" > secrets/buildbot_www_pass.txt
+$ export HALIDE_BB_MASTER_ARTIFACTS_DIR=/srv/www/buildbot/public_html/artifacts
 ```
 
-Then, create a database for the master to save its work. This only needs to be
-done once.
+Then start all services:
 
 ```console
-$ ./master.sh upgrade-master
+$ docker compose up -d --build
 ```
 
-Choose a directory to hold artifacts for package runs:
+The database is automatically initialized on first start. Caddy handles TLS
+certificates, so there is no manual web server configuration needed. The
+Buildbot web interface is available at `https://buildbot.halide-lang.org/master/`
+and artifacts are served at the site root.
+
+Port 9990 must be reachable by workers. Port 8012 is internal only; Caddy
+proxies it on ports 80/443.
+
+### Running without Docker
+
+By default, the master uses a SQLite database (`sqlite:///state.sqlite`), so
+no external database is required. To use PostgreSQL instead, set
+`HALIDE_BB_MASTER_DB_URL`:
 
 ```console
-$ export HALIDE_BB_MASTER_ARTIFACTS_DIR=/srv/www/buildbot/public_html/artifacts
+$ export HALIDE_BB_MASTER_DB_URL="postgresql://buildbot:{DB_PASSWORD}@db/buildbot"
 ```
 
-Finally, start the master!
+Then start the master:
 
 ```console
 $ ./master.sh start
 ```
 
+The script automatically runs `upgrade-master` before starting. If the master
+is already running, invoking `./master.sh` with no arguments will reconfig it
+instead.
+
 # Worker configuration
 
 The master recognizes workers by their reported names, e.g. `linux-worker-4`
-or `win-worker-1`. To launch the buildbot daemon on the worker named
-`$WORKER_NAME`, run the following commands after setting up the Python
-environment as detailed above:
+or `win-worker-1`.
+
+## Linux / macOS
+
+Write the worker secret and set the worker name, then use the wrapper script:
 
 ```console
 $ echo "$WORKER_SECRET" > worker/halide_bb_pass.txt
-$ export HALIDE_BB_WORKER_NAME=$WORKER_NAME  # required
-$ export HALIDE_BB_MASTER_ADDR=$MASTER_ADDR  # default = public Halide master
-$ export HALIDE_BB_MASTER_PORT=$MASTER_PORT  # default = 9990
-$ uv run --package worker buildbot-worker start worker
+$ export HALIDE_BB_WORKER_NAME=$WORKER_NAME
+$ ./worker.sh
 ```
 
-[uv]: https://docs.astral.sh/uv
+## Windows
+
+```powershell
+> Set-Content worker/halide_bb_pass.txt "$WORKER_SECRET"
+> $env:HALIDE_BB_WORKER_NAME = "$WORKER_NAME"
+> .\worker.ps1
+```
+
+## Optional environment variables
+
+| Variable                | Default                    | Description     |
+|-------------------------|----------------------------|-----------------|
+| `HALIDE_BB_MASTER_ADDR` | `buildbot.halide-lang.dev` | Master hostname |
+| `HALIDE_BB_MASTER_PORT` | `9990`                     | Master PB port  |
+
+## Platform-specific installation
+
+Automated installation scripts that set up system dependencies and configure
+the worker to start automatically are provided under `worker/`:
+
+- **macOS:** `worker/macos/install.sh` — installs Homebrew dependencies,
+  configures ccache, and installs a launchd agent so the worker starts on login.
+  Requires `HALIDE_BB_WORKER_NAME`, `HL_WEBGPU_NODE_BINDINGS`,
+  `HL_WEBGPU_NATIVE_LIB`, and `EMSDK` to be set before running.
+
+- **Windows:** `worker/windows/install.ps1` — installs dependencies via winget,
+  sets up Visual Studio 2022, installs uv, and bootstraps vcpkg with the
+  required libraries.
+
+[Caddy]: https://caddyserver.com
+[dc]: https://docs.docker.com/compose/
+[uv]: https://docs.astral.sh/uv

docker-compose.yml

@@ -9,6 +9,8 @@ services:
       POSTGRES_DB: buildbot
       POSTGRES_USER: buildbot
       POSTGRES_PASSWORD_FILE: /run/secrets/db_password.txt
+    logging:
+      driver: local
     restart: unless-stopped
 
   buildbot:
@@ -31,6 +33,9 @@ services:
     environment:
       HALIDE_BB_MASTER_ARTIFACTS_DIR: /artifacts
       HALIDE_BB_MASTER_SECRETS_DIR: /run/secrets
+      HALIDE_BB_MASTER_DB_URL: "postgresql://buildbot:{DB_PASSWORD}@db/buildbot"
+    logging:
+      driver: local
     restart: unless-stopped
 
   caddy:
@@ -44,6 +49,8 @@ services:
       - caddy-data:/data
       - caddy-config:/config
       - ${HALIDE_BB_MASTER_ARTIFACTS_DIR:-./data/artifacts}:/artifacts:ro
+    logging:
+      driver: local
     restart: unless-stopped
 
 volumes:

master.sh

@@ -28,12 +28,19 @@ fi
 # Check necessary files are present
 
 secrets_dir="${HALIDE_BB_MASTER_SECRETS_DIR:-secrets}"
-for secret in buildbot_www_pass db_password github_token halide_bb_pass webhook_token; do
+for secret in buildbot_www_pass github_token halide_bb_pass webhook_token; do
   if [ ! -s "$secrets_dir/${secret}.txt" ]; then
     fail "Missing or empty $secrets_dir/${secret}.txt: cannot continue"
   fi
 done
 
+db_url="${HALIDE_BB_MASTER_DB_URL:-sqlite:///state.sqlite}"
+if echo "$db_url" | grep -q '{DB_PASSWORD}'; then
+  if [ ! -s "$secrets_dir/db_password.txt" ]; then
+    fail "Missing or empty $secrets_dir/db_password.txt: required by HALIDE_BB_MASTER_DB_URL"
+  fi
+fi
+
 ##
 # Resolve the buildbot command
 

master/Dockerfile

@@ -21,26 +21,17 @@ RUN apt-get update \
 # Non-root user
 RUN useradd --create-home --home-dir /buildbot buildbot
 
+USER buildbot
+WORKDIR /buildbot
+
 # Copy virtual environment from builder
-COPY --from=builder /buildbot/.venv /buildbot/.venv
+COPY --from=builder --chown=buildbot:buildbot /buildbot/.venv /buildbot/.venv
 ENV PATH="/buildbot/.venv/bin:$PATH" \
     VIRTUAL_ENV="/buildbot/.venv"
 
-# Copy master source files
-COPY master/master.cfg \
-     master/custom_steps.py \
-     master/buildbot.tac \
-     master/toolchain.linux-arm32.cmake \
-     /buildbot/master/
-
-COPY master.sh /buildbot/master.sh
-RUN chmod +x /buildbot/master.sh
-
-# Ensure buildbot owns its home directory
-RUN chown -R buildbot:buildbot /buildbot
-
-USER buildbot
-WORKDIR /buildbot
+# Copy source files
+COPY --chown=buildbot:buildbot master/ master/
+COPY --chown=buildbot:buildbot master.sh master.sh
 
 EXPOSE 8012 9990
 

master/buildbot.tac

@@ -1,29 +1,14 @@
-import os
+import sys
 from pathlib import Path
 
 from buildbot.master import BuildMaster
 from twisted.application import service
-from twisted.python.log import ILogObserver, FileLogObserver
-from twisted.python.logfile import LogFile
-
-basedir = str(Path(__file__).parent.resolve())
-rotateLength = 10000000
-maxRotatedFiles = 10
-configfile = "master.cfg"
-
-# Default umask for server
-umask = None
-
-logfile = LogFile.fromFullPath(
-    os.path.join(basedir, "twistd.log"), rotateLength=rotateLength, maxRotatedFiles=maxRotatedFiles
-)
+from twisted.logger import ILogObserver, textFileLogObserver
 
 # note: this line is matched against to check that this is a buildmaster
 # directory; do not edit it.
 application = service.Application('buildmaster')  # fmt: skip
-application.setComponent(ILogObserver, FileLogObserver(logfile).emit)
+application.setComponent(ILogObserver, textFileLogObserver(sys.stdout))
 
-m = BuildMaster(basedir, configfile, umask)
+m = BuildMaster(str(Path(__file__).parent), "master.cfg", umask=None)
 m.setServiceParent(application)
-m.log_rotation.rotateLength = rotateLength
-m.log_rotation.maxRotatedFiles = maxRotatedFiles

master/master.cfg

@@ -52,11 +52,14 @@ SECRETS_DIR = REPO_DIR / os.environ.get("HALIDE_BB_MASTER_SECRETS_DIR", "secrets
 # SECRETS
 
 BUILDBOT_WWW_PASS = (SECRETS_DIR / "buildbot_www_pass.txt").read_text().strip()
-DB_PASSWORD = (SECRETS_DIR / "db_password.txt").read_text().strip()
 GITHUB_TOKEN = (SECRETS_DIR / "github_token.txt").read_text().strip()
 HALIDE_BB_PASS = (SECRETS_DIR / "halide_bb_pass.txt").read_text().strip()
 WEBHOOK_TOKEN = (SECRETS_DIR / "webhook_token.txt").read_text().strip()
 
+DB_URL = os.environ.get("HALIDE_BB_MASTER_DB_URL", "sqlite:///state.sqlite")
+if "{DB_PASSWORD}" in DB_URL:
+    DB_PASSWORD = (SECRETS_DIR / "db_password.txt").read_text().strip()
+    DB_URL = DB_URL.replace("{DB_PASSWORD}", DB_PASSWORD)
 
 # LLVM
 
@@ -2200,7 +2203,7 @@ c["buildbotURL"] = "https://buildbot.halide-lang.org/master/"
 # DB URL
 
 c["db"] = {
-    "db_url": f"postgresql://buildbot:{DB_PASSWORD}@db/buildbot",
+    "db_url": DB_URL,
 }
 
 # GitHub Integration