dhuseby/refactor/perf-and-transport-and-hole-punch

[dhuseby]

This is a major refactor of the whole system. I did not touch any of the gossipsub testing though, just perf, hole-punch, and transport.

Key improvements:

• Everything is POSIX bash
• All variables are properly/safely used (e.g. "${FOO}")
• All common functions moved to scripts in the lib/ subdir
• Test matrix generation is now sharded and parallel with no disk I/O in the nested loops
• Test execution is parallel (except perf, it forces 1 worker on purpose) with common global services (i.e. Redis) started before and shutdown after all of the individual tests
• Test folders only hold test-specific data and functions. This significantly simplifies tests.
• All test applications take a common set of environment variables plus test-specific environment variables.
• All test applications output YAML results to a per-test results file
• Added the ability to patch a remote test application before building the docker image. This enables quick debugging and iteration and landing the update in test-plans BEFORE the fixes land in the implementation repo
• All three tests have a common structure with very similary code. The per-test scripts are also very similar with a common set of steps
• Clean, uniform formatting of output
• Better debug output when --debug is passed as an option. This also sets the DEBUG environment variable for test applications to respond to

[dhuseby]

The perf and transport tests are working. Test applications need to be updated. See the docs/write-a-perf-test-app.md and docs/write-a-transport-test-app.md files to see how to do it.

I also added patching of remote test applications. See the transport/images/rust/v0.56/transport-fix.patch and the 'rust-v0.56' test image definition in the transport/images.yaml file to see how that is done.

The transport/README.md is up to date if you want to know more. The run.sh scripts in each test folder are pretty well documented as are the scripts in the lib/ folder. I will be fully documenting all of this and writing a blog post explaining everything.

[dhuseby]

The breaking changes I made are in the interest of normalizing the interface to each test app, regardless of test. The common environment variables are the same for every test with additional test-specific environment variables being set as well. The expected output from the test application is in YAML now so that comments can be added if desired.

I'm measuring a significant speed-up in test matrix generation. The transport test will generate something like ~1200 unique tests. Running the test-matrix.yaml generation with sharding and parallel processing, without any I/O in the nested loops, yields a massive increase in performance. On an 8-year old 4-core machine, it takes ~27 seconds to generate the full matrix from scratch. It used to take 4 to 5 minutes.

The same is true with the parallel execution of isolated tests with global services and name-spacing. Significant performance gain.

[dhuseby]

Not quite ready yet. Hole-punching cleanup needs completing.

[dhuseby]

alright...it's mostly done, the only thing left is to fix up the test applications to match the new uniform "contract" between the test framework and the test applications. perf, transport, and hole-punch tests all run. i fixed the baseline tests. documented everything, removed unused code. i wrote a peer and relay in rust for the hole-punch tests but it doesn't seem to work.

This is ready for a review.

[seetadev]

@dhuseby : This is excellent work — thanks for pushing this through 👍
The scope and depth of the refactor really show, and the normalization of the test app contract, parallelization, and cleanup are all big wins for maintainability and performance.

We’re already reviewing this in parallel, and I’ll also walk through it in detail with contributors in today’s maintainer call so we can align on the breaking changes and the path to updating the test applications. The documentation you added around the new contract and the patching workflow for remote test apps is especially helpful and should make the transition much smoother for folks.

The performance improvements you’re seeing in matrix generation and execution are impressive, and the uniform YAML output + shared structure across perf, transport, and hole-punch tests feels like the right long-term direction. Great call on prioritizing correctness and consistency over backwards compatibility here.

Thanks again for the thorough cleanup and for flagging the remaining hole-punch nuances clearly. Looking forward to giving more concrete review feedback after the maintainer discussion.

[dhuseby]

I was trying to answer a question about a specific bug and I realized that negative filtering wasn't enough so I added the positive filtering back in. Also, I added --impl-select and --impl-ignore to mean implementation ids (e.g. rust-v0.56, etc) and repurposed the --test-select and --test-ignore to be filters applied to the actual name of each test. So for instance, if you want to run just tests where rust-v0.56 dials nim-v1.14, you can do ./run.sh --test-select 'rust-v0.56 x nim-v1.14' and that will run only the tests with rust as the dialer and nim as the listener. (watch out though because this is a substring match and this exact example also matches tests with the name chromium-rust-v0.56 x nim-v1.14 so if you want to not include those browser tests, then also add --impl-ignore 'chromium')

All of the docs are updated. All three tests are updated and tested. I think this is the last round or revisions for this PR.