multi-server: refactor test framework for parameterised tests

Restructure the multi-server test framework so that each test is
defined by a single YAML params file.  All templates (compose,
radiusd configs, test steps) are rendered from shared Jinja2
templates using those params.

- Replace monolithic all.mk with macro-based framework that
  auto-discovers test suites and param files
- Move configs to configs/freeradius/, environments to environments/
- Rename test suites to proxy-accept and proxy-multihop-accept
- Use *.test.yml for param files, *.ci.test.yml for CI-only tests
- Add test.multi-server.ci target for short CI tests
- Render all build products into build/ directory
- Support parallel execution with unique Docker project names
- Add healthchecks to compose templates
- Remove hardcoded port mappings (containers use internal networking)
- Add CI workflow for GitHub Actions with Docker-in-Docker

Author: arr2036

.github/workflows/ci-multi-server-tests.yml

@@ -0,0 +1,100 @@
+name: Multi-Server CI Tests
+
+on:
+  push:
+    branches-ignore:
+      - coverity_scan
+      - run-fuzzer**
+      - debug-fuzzer-**
+  pull_request:
+    paths:
+      - '.github/workflows/ci-multi-server-tests.yml'
+      - 'src/tests/multi-server/**'
+  workflow_dispatch:
+
+jobs:
+  multi-server-tests:
+    runs-on: self-hosted
+
+    services:
+      dind:
+        image: docker:dind
+        options: --privileged
+        env:
+          DOCKER_TLS_CERTDIR: ""
+          #  Bypass the squid proxy for internal registry access.
+          NO_PROXY: "*.networkradius.com,127.0.0.1"
+        #  Mount the host's internal CA so dind trusts
+        #  docker.internal.networkradius.com for image pulls.
+        #
+        #  Share the runner's workspace with dind so that docker
+        #  compose bind-mounts (radiusd.conf, env-setup.sh, listener
+        #  dirs, etc.) resolve to real files inside the dind daemon.
+        #
+        #  github.workspace is the HOST path to the workspace.
+        #  The runner mounts it into the job container at a
+        #  different path (/__w/...), so we use a fixed mount
+        #  point (/workspace) that both containers agree on.
+        volumes:
+          - /usr/local/share/ca-certificates/networkradius.com.crt:/etc/docker/certs.d/docker.internal.networkradius.com/ca.crt:ro
+          - ${{ github.workspace }}:/workspace
+
+    container:
+      image: docker.internal.networkradius.com/self-hosted
+      #  "privileged" is needed for Samba install
+      #  "memory-swap -1" enables full use of host swap and may help
+      #    with containers randomly quitting with "The operation was
+      #    canceled"
+      options: >-
+        --privileged
+        --memory-swap -1
+      env:
+        DOCKER_HOST: tcp://dind:2375
+        NO_PROXY: dind
+      #  Shared workspace — see dind volumes comment above.
+      volumes:
+        - ${{ github.workspace }}:/workspace
+
+    defaults:
+      run:
+        working-directory: /workspace
+
+    steps:
+
+      - name: Install extra packages
+        run: |
+          apt-get update && apt-get install -y --no-install-recommends docker.io docker-buildx docker-compose-v2 python3-venv
+
+      - uses: actions/checkout@v4
+        with:
+          lfs: false
+
+      #  Authenticate to the internal registry via the dind daemon.
+      #  The host Docker daemon is logged in via the runner's
+      #  job-started hook, but dind is a separate daemon with no
+      #  auth config.
+      - name: Login to internal Docker registry
+        env:
+          DOCKER_USERNAME: ${{ secrets.DOCKER_REPO_USERNAME }}
+          DOCKER_PASSWORD: ${{ secrets.DOCKER_REPO_PASSWORD }}
+        run: |
+          echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin https://docker.internal.networkradius.com/
+
+      - name: Build Docker image from source
+        run: |
+          make docker.ubuntu24.build
+          docker tag freeradius4/ubuntu24:latest freeradius-build:latest
+
+      - name: Run multi-server tests
+        run: |
+          make -j$(nproc) test.multi-server.ci
+
+      #
+      #  If the CI has failed and the branch is ci-debug
+      #  then start a tmate session for interactive debugging.
+      #
+      - name: "Debug: Start tmate"
+        uses: mxschmitt/action-tmate@v3
+        with:
+          limit-access-to-actor: true
+        if: ${{ github.ref == 'refs/heads/ci-debug' && failure() }}

Makefile

@@ -43,7 +43,9 @@ endif
 #  there's no point in requiring the developer to run configure
 #  *before* making packages.
 #
-ifeq "$(filter deb rpm pkg_version dist-check% crossbuild.% docker.% freeradius-server-%,$(MAKECMDGOALS))" ""
+#  Multi-server tests use Docker and don't need a local build.
+#
+ifeq "$(filter deb rpm pkg_version dist-check% crossbuild.% docker.% freeradius-server-% test.multi-server%,$(MAKECMDGOALS))" ""
   $(if $(wildcard Make.inc),,$(error Missing 'Make.inc' Run './configure [options]' and retry))
   include Make.inc
 else
@@ -112,10 +114,10 @@ PROTOCOLS    := \
 	tls
 
 #
-#  If we're building packages or crossbuilding, just do that.
-#  Don't try to do a local build.
+#  If we're building packages, crossbuilding, or running Docker-based
+#  tests, just do that.  Don't try to do a local build.
 #
-ifeq "$(filter deb rpm pkg_version dist-check% crossbuild.% docker.% freeradius-server-%,$(MAKECMDGOALS))" ""
+ifeq "$(filter deb rpm pkg_version dist-check% crossbuild.% docker.% freeradius-server-% test.multi-server%,$(MAKECMDGOALS))" ""
 
 #
 #  Include all of the autoconf definitions into the Make variable space
@@ -575,3 +577,10 @@ endif
 ifneq "$(findstring docker,$(MAKECMDGOALS))" ""
   include scripts/docker/docker.mk
 endif
+
+#
+#  Multi-server integration tests (Docker-based, no configure needed)
+#
+ifneq "$(findstring test.multi-server,$(MAKECMDGOALS))" ""
+  include src/tests/multi-server/all.mk
+endif

src/tests/multi-server/README.md

@@ -1,98 +1,82 @@
-If you are reading this, you are probably wondering how to run a multi-server test.  Here's a quick overview.
+# Multi-Server Tests
 
-## Before You Begin
+Integration tests that spin up multiple FreeRADIUS instances in Docker
+containers and verify they can proxy traffic between each other.
 
-Running the multi-server tests requires the availability of a Docker
-image `freeradius-build:latest` to be available on the host running
-the tests.
+## Prerequisites
+
+A `freeradius-build:latest` Docker image must be available locally:
 
 ```bash
-% cd ${FREERADIUS-SERVER-LOCAL-REPO}
-% make docker.ubuntu24.build
-% docker tag <your-freeradius-docker-image-tag> freeradius-build:latest
+make docker.ubuntu24.build
+docker tag freeradius4/ubuntu24:latest freeradius-build:latest
 ```
 
-## Run Test With Makefile
-
-### Run All Tests
+## Running Tests
 
-2. Run make target based on the test name. All testcase config files
-start with "test-*".
+### All tests
 
 ```bash
-% make -f src/tests/multi-server/all.mk
+make test.multi-server
 ```
 
-### Run Specific Tests (Example)
+### CI tests only (short tests)
 
 ```bash
-% make -f src/tests/multi-server/all.mk test-5hs-autoaccept
+make test.multi-server.ci
 ```
 
-or
+### A specific test
 
 ```bash
-% make -f src/tests/multi-server/all.mk test-1p-2hs-autoaccept
+make test.multi-server.proxy-accept.short.ci
 ```
 
-### Optional Debug Logs and Logging Verbosity Level
+### Parallel execution
 
 ```bash
-% make -f src/tests/multi-server/all.mk test-5hs-autoaccept DEBUG=1 VERBOSE=4
+make -j$(nproc) test.multi-server
 ```
 
-## Run Multi-Server Tests Manually Without Makefile (Optional)
+### Extra flags
 
-### Clone Multi-Server Test Framework Repo & Activate Python venv
+Pass debug/verbosity flags to the test framework:
 
 ```bash
-cd src/tests/multi-server/
-git clone git@github.com:InkbridgeNetworks/freeradius-multi-server.git 
-cd freeradius-multi-server
-./configure
-source .venv/bin/activate
+make test.multi-server TEST_MULTI_SERVER_FLAGS="-xx -vvv"
 ```
 
-### Render Jinja Templates
+## How It Works
 
-In this example we render the Jinja templates used by the environment
-configuration used by the `test-5hs-autoaccept` test.
+Each test suite is a directory under `tests/` containing:
 
-The Docker Compose `env-5hs-autoaccept.yml` file represents the
-`load-generator -> 5 homeserver` test environment used by the test.
+- `template.yml.j2` - Jinja2 template for test steps (state machine)
+- `environment.yml.j2` - Symlink to a Docker Compose template in `environments/`
+- `*.test.yml` - Parameter files (one per test variant)
 
-Render FreeRADIUS "homeserver" Virtual Server config:
+A parameter file is flat YAML defining topology, load profile, and test
+timeouts.  All `.j2` files in the suite directory are rendered using
+these parameters.  The rendered compose file's `${DATA_PATH}` volume
+mounts are scanned and the corresponding config files are copied or
+rendered into the build directory.
 
-```bash
-% python3 src/config_builder.py \
-    --vars-file "${top_srcdir}/src/tests/multi-server/environments/jinja-vars/env-5hs-autoaccept.vars.yml" \
-    --aux-file "${top_srcdir}/src/tests/multi-server/environments/configs/freeradius/homeserver/radiusd.conf.j2" \
-    --include-path "${top_srcdir}/src/tests/multi-server/"
- ```
- Render FreeRADIUS "load-generator" Virtual Server config:
- ```bash
- python3 src/config_builder.py \
-    --vars-file "${top_srcdir}/src/tests/multi-server/environments/jinja-vars/env-5hs-autoaccept.vars.yml" \
-    --aux-file "${top_srcdir}/src/tests/multi-server/environments/configs/freeradius/load-generator/radiusd.conf.j2" \
-    --include-path "${top_srcdir}/src/tests/multi-server/"
-```
+Build outputs go to `build/tests/multi-server/<suite>/<test>/`.
 
-Render Docker Compose environment:
+## Adding a New Test
 
-```bash
- python3 src/config_builder.py \
-    --vars-file "${top_srcdir}/src/tests/multi-server/environments/jinja-vars/env-5hs-autoaccept.vars.yml" \
-    --aux-file "${top_srcdir}/src/tests/multi-server/environments/docker-compose/env-5hs-autoaccept.yml.j2" \
-    --include-path "${top_srcdir}/src/tests/multi-server/"
-```
+1. Create a parameter file in an existing suite directory, e.g.
+   `tests/proxy-accept/heavy.test.yml`.  Name it `*.ci.test.yml` if
+   it should run in CI.
 
-### Run the test:
+2. Or create a new suite directory with `template.yml.j2`,
+   `environment.yml.j2` (symlink), and parameter files.
 
-```bash
-% DATA_PATH="${top_srcdir}/src/tests/multi-server/environments/configs" \
-			make test-framework -- -x -v \
-			--compose "${top_srcdir}/src/tests/multi-server/environments/docker-compose/env-5hs-autoaccept.yml" \
-			--test "${top_srcdir}/src/tests/multi-server/test-5hs-autoaccept.yml" \
-			--use-files \
-			--listener-dir "${top_srcdir}/build/tests/multi-server/freeradius-multi-server-test-runtime-logs/test-5hs-autoaccept"
-```
+The build framework discovers suites automatically by finding
+directories containing `template.yml.j2`, and discovers tests by
+finding `*.test.yml` files within them.
+
+## File Naming Conventions
+
+- `*.test.yml` - Test parameter file (discovered by `make test.multi-server`)
+- `*.ci.test.yml` - CI test parameter file (also discovered by `make test.multi-server.ci`)
+- `*.yml.j2` - Jinja2 template (rendered, not treated as a test)