add support for testing Webtransport by marten-seemann · Pull Request #464 · quic-interop/quic-interop-runner

marten-seemann · 2026-02-28T12:26:05Z

For now this just works on the command line. The GitHub Actions workflow and the website will be extended in a later PR.

Copilot

Pull request overview

Adds initial command-line support for running WebTransport interoperability tests alongside existing QUIC tests, including new protocol docs, test cases, and runner plumbing to select protocol-specific implementations and request semantics.

Changes:

Introduces WebTransport protocol documentation and a new WebTransport test suite (handshake + transfer variants).
Refactors test framework to support separate client/server www & downloads mounts and per-side request lists.
Adds protocol selection (--protocol) and splits implementation registries into QUIC vs WebTransport JSON files.

Reviewed changes

Copilot reviewed 13 out of 15 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
webtransport.md	Adds WebTransport protocol + testcase specification for endpoints.
testcases_webtransport.py	Implements WebTransport handshake/transfer testcases and verification logic.
testcases.py	Refactors QUIC testcases to use new shared `testcase.TestCase` base and “raw paths” abstraction.
testcase.py	Introduces shared base TestCase with separate client/server dirs, file generation, and trace helpers.
run.py	Adds `--protocol` selection and wires protocol-specific implementations + testcase lists.
quic.md	Adds QUIC-specific documentation moved out of README.
pull.py	Adds `--protocol` selection for pulling QUIC vs WebTransport implementation images.
interop.py	Updates runner to pass separate mounts/REQUESTS/protocols for client and server containers.
implementations_webtransport.json	Adds WebTransport implementation registry.
implementations.py	Replaces global `IMPLEMENTATIONS` with loader functions for per-protocol registries.
empty.env	Adds protocol env defaults for compose.
docker-compose.yml	Switches to separate client/server www/download mounts and per-side REQUESTS/PROTOCOLS mapping.
README.md	Updates docs for QUIC/WebTransport split and new CLI usage.
.gitignore	Stops ignoring protocol-specific implementation registries.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

testcases_webtransport.py

Copilot · 2026-02-28T12:32:04Z

run.py

-    for name, value in IMPLEMENTATIONS.items()
-    if value["role"] == Role.BOTH or value["role"] == Role.SERVER
-]
+from testcases_quic import MEASUREMENTS, TESTCASES_QUIC


This imports testcases_quic, but in this PR diff the QUIC testcases live in testcases.py (and no testcases_quic.py file is shown). This will raise ModuleNotFoundError at runtime unless testcases_quic.py is added/renamed accordingly. Align the module name (either add/rename the file or adjust imports back to the actual module).

Suggested change

from testcases_quic import MEASUREMENTS, TESTCASES_QUIC

from testcases import MEASUREMENTS, TESTCASES_QUIC

testcase.py

testcases_webtransport.py

testcase.py

webtransport.md

Copilot

Pull request overview

Copilot reviewed 13 out of 15 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

interop.py

implementations.py

Copilot · 2026-03-01T02:43:54Z

run.py

            "--test",
            help="test cases (comma-separatated). Valid test cases are: "
-            + ", ".join([x.name() for x in TESTCASES + MEASUREMENTS]),
+            + ", ".join([x.name() for x in TESTCASES_QUIC + MEASUREMENTS]),


The --test help text always lists QUIC testcases (TESTCASES_QUIC + MEASUREMENTS), even when --protocol webtransport is selected. This makes the CLI misleading. Consider listing protocol-agnostic test names (union of both sets), or parsing --protocol first and then constructing the parser/help accordingly.

Suggested change

+ ", ".join([x.name() for x in TESTCASES_QUIC + MEASUREMENTS]),

+ ", ".join(

sorted(

{

x.name()

for x in TESTCASES_QUIC + MEASUREMENTS + TESTCASES_WEBTRANSPORT

}

)

),

Copilot · 2026-03-01T02:43:54Z

run.py

        if arg is None:
-            return TESTCASES, MEASUREMENTS
+            return testcases, MEASUREMENTS
        elif arg == "onlyTests":
-            return TESTCASES, []
+            return testcases, []
        elif arg == "onlyMeasurements":


When --protocol webtransport is selected and -t/--test is not provided, this returns MEASUREMENTS from the QUIC suite by default. That will attempt to run QUIC measurements against WebTransport implementations, which is very likely incorrect/noisy. Consider returning an empty measurement list for WebTransport (or introduce MEASUREMENTS_WEBTRANSPORT).

So far, the server had only /www and the client had only /downloads.

Copilot

Pull request overview

Copilot reviewed 13 out of 15 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-01T04:49:29Z

run.py

-            return [], MEASUREMENTS
+            return [], measurements
        elif not arg:
            return []


get_tests_and_measurements() sometimes returns a bare list (return []) when arg is an empty string, but callers treat the return value as a 2-tuple (t[0], t[1]). This will raise at runtime. Return a (tests, measurements) tuple in all branches (e.g., ([], []) here).

Suggested change

return []

return [], []

Copilot · 2026-03-01T04:49:30Z

run.py

+    protocol = get_args().protocol
+    if protocol == "quic":
+        impls = get_quic_implementations()
+    elif protocol == "webtransport":
+        impls = get_webtransport_implementations()
+    else:
+        sys.exit("Unknown protocol: " + protocol)


get_args() / parse_args() is called repeatedly throughout main(). This re-parses argv multiple times and makes it harder to reason about which arguments are actually in effect. Parse once (e.g., args = get_args()) and reference args.<field> thereafter.

Copilot · 2026-03-01T04:49:30Z

empty.env

 WWW=""
 WAITFORSERVER=""
+PROTOCOLS_CLIENT=""
+PROTOCOLS_SERVER=""


empty.env is used with docker compose --env-file empty.env (e.g., for stop). Since docker-compose.yml now interpolates CLIENT_WWW, CLIENT_DOWNLOADS, SERVER_WWW, SERVER_DOWNLOADS, REQUESTS_CLIENT, and REQUESTS_SERVER, consider adding them here (even as empty defaults) to avoid noisy "variable is not set" warnings and ensure consistent behavior across Compose versions.

Suggested change

PROTOCOLS_SERVER=""

PROTOCOLS_SERVER=""

CLIENT_WWW=""

CLIENT_DOWNLOADS=""

SERVER_WWW=""

SERVER_DOWNLOADS=""

REQUESTS_CLIENT=""

REQUESTS_SERVER=""

Copilot · 2026-03-01T04:49:30Z

webtransport.md

+
+## Testcases
+
+**Handshake** (`handshake`): Tests the successful completion of the handshake. Both client and server save the negotiated protocol in a file called `negotiated_protocol.txt` in the directory specified by the `DOWNLOAD_DIR` environment variable.


The Handshake testcase description references a DOWNLOAD_DIR environment variable, but the runner mounts a fixed /downloads/ directory into containers (and doesn't set DOWNLOAD_DIR). Update this to point to /downloads/ (or document the actual env var if one is intended) so implementers know where to write negotiated_protocol.txt.

Suggested change

**Handshake** (`handshake`): Tests the successful completion of the handshake. Both client and server save the negotiated protocol in a file called `negotiated_protocol.txt` in the directory specified by the `DOWNLOAD_DIR` environment variable.

**Handshake** (`handshake`): Tests the successful completion of the handshake. Both client and server save the negotiated protocol in a file called `negotiated_protocol.txt` in the `/downloads/` directory.

marten-seemann requested a review from Copilot February 28, 2026 12:26

Copilot AI reviewed Feb 28, 2026

View reviewed changes

marten-seemann force-pushed the webtransport branch 3 times, most recently from bf58d08 to ca6e374 Compare February 28, 2026 13:18

Copilot started reviewing on behalf of marten-seemann February 28, 2026 15:20 View session

marten-seemann force-pushed the webtransport branch 2 times, most recently from 91ff03e to c5b1b63 Compare March 1, 2026 02:38

marten-seemann requested a review from Copilot March 1, 2026 02:38

Copilot started reviewing on behalf of marten-seemann March 1, 2026 02:39 View session

Copilot AI reviewed Mar 1, 2026

View reviewed changes

marten-seemann added 2 commits March 1, 2026 11:06

generalize setup to allow for testing of other protocols

a8561b2

mount a www and a downloads directory into both client and server

453a686

So far, the server had only /www and the client had only /downloads.

marten-seemann force-pushed the webtransport branch from c5b1b63 to cb4caec Compare March 1, 2026 03:06

marten-seemann added 4 commits March 1, 2026 12:35

add WebTransport handshake, stream and datagram tests

804c38c

add WebTransport test specification

10a86b3

include WebTransport test cases in cli help output

fc9b0cb

don’t output a measurement table for WebTransport

b47bd32

marten-seemann force-pushed the webtransport branch from 1f7da71 to b47bd32 Compare March 1, 2026 04:42

marten-seemann requested a review from Copilot March 1, 2026 04:44

Copilot started reviewing on behalf of marten-seemann March 1, 2026 04:45 View session

Copilot AI reviewed Mar 1, 2026

View reviewed changes

	from testcases_quic import MEASUREMENTS, TESTCASES_QUIC
	from testcases import MEASUREMENTS, TESTCASES_QUIC

-PROTOCOLS_SERVER=""
+PROTOCOLS_SERVER=""
+CLIENT_WWW=""
+CLIENT_DOWNLOADS=""
+SERVER_WWW=""
+SERVER_DOWNLOADS=""
+REQUESTS_CLIENT=""
+REQUESTS_SERVER=""


		## Testcases

		Handshake (`handshake`): Tests the successful completion of the handshake. Both client and server save the negotiated protocol in a file called `negotiated_protocol.txt` in the directory specified by the `DOWNLOAD_DIR` environment variable.

Uh oh!

Conversation

marten-seemann commented Feb 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants