New transport interop runner changed test spec

[MarcoPolo]

The original test spec for the transport interop used lowercase env parameters: https://github.com/libp2p/test-plans/tree/555850b45d1c0c45f2004a8999dde0426da72737/transport-interop#test-spec.

The new version uses upper case env parameters. This breaks all existing transport interop implementations.

There's no reason for this breakage. Using lower case env vars is in line with the posix spec

The name space of environment variable names containing lowercase letters is reserved for applications.

We shouldn't break the implementations for no good reason.

For a similar reason, we shouldn't change the output format required from the dialer.

[dhuseby]

It has been convention to use all upper case environment variables since the early 1980's. This change is to make this test suite aligned with expectations of developers and lower the barrier to entry.

The fix for all of the test applications is trivial and is being addressed already and will be completed soon.

I don't understand why you're so concerned with something that improves the test framework and causes a very temporary breakage. Your preference isn't aligned with well-established convention.

[MarcoPolo]

To be clear, I don't have a strong opinion about the details of upper vs lower case or yaml vs json, I do care about not breaking backwards compatibility.

[dhuseby]

I'm glad to hear you don't have a strong opinion either way. The current breaks will be fixed ASAP. It's important to point out that none of these tests were working at all when I started the refactor in November. Everything that was changed in this rewrite was done for specific reasons in pursuit of better developer ergonomics and velocity to support the research projects and spec work.

A couple of updates for you:

1. We have a new website in the works that will be launching soon. I would love feedback/broken link reports. You can see it here: https://libp2p.myfilebase.site
2. In addition to the new site, I added a blog post on the test-plans rewrite here: https://libp2p.myfilebase.site/blog/new-test-plans/
3. I also added four new guides on testing:
   3.1. https://libp2p.myfilebase.site/guides/local-testing-strategies/
   3.2. https://libp2p.myfilebase.site/guides/write-a-transport-test-app/
   3.3. https://libp2p.myfilebase.site/guides/write-a-perf-test-app/
   3.4. https://libp2p.myfilebase.site/guides/write-a-hole-punch-test-app/

If anything is not clear, let me know. Happy to address them. We're pushing forward and will have all of the broken tests back up and running very soon. The go tests are being addressed by @acul71: #801 I'll have all of the rust ones patched up by the end of the week and there are others addressing the JS/Nim/Py/Zig/C++/Dotnet patches.

[MarcoPolo]

None of these tests were working at all when I started the refactor in November.

This is false. I just picked up commit f58b747 from right before November. It works fine on a fresh VM[1] with docker compose. I built the implementation images, ran npm ci, then ran npm run test.

Did you see issues with podman? I noticed it parses the compose yaml a bit differently and sets is_dialer=True rather than is_dialer=true.

[1]: GitHub Codespaces: Linux codespaces-999e14 6.8.0-1030-azure #35~22.04.1-Ubuntu SMP Mon May 26 18:08:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

[jxs]

I'm glad to hear you don't have a strong opinion either way. The current breaks will be fixed ASAP. It's important to point out that none of these tests were working at all when I started the refactor in November. Everything that was changed in this rewrite was done for specific reasons in pursuit of better developer ergonomics and velocity to support the research projects and spec work.

A couple of updates for you:

1. We have a new website in the works that will be launching soon. I would love feedback/broken link reports. You can see it here: https://libp2p.myfilebase.site
2. In addition to the new site, I added a blog post on the test-plans rewrite here: https://libp2p.myfilebase.site/blog/new-test-plans/
3. I also added four new guides on testing:
   3.1. https://libp2p.myfilebase.site/guides/local-testing-strategies/
   3.2. https://libp2p.myfilebase.site/guides/write-a-transport-test-app/
   3.3. https://libp2p.myfilebase.site/guides/write-a-perf-test-app/
   3.4. https://libp2p.myfilebase.site/guides/write-a-hole-punch-test-app/

If anything is not clear, let me know. Happy to address them. We're pushing forward and will have all of the broken tests back up and running very soon. The go tests are being addressed by @acul71: #801 I'll have all of the rust ones patched up by the end of the week and there are others addressing the JS/Nim/Py/Zig/C++/Dotnet patches.

Hi Dave, the motives may all be valid, but #773 shouldn't have been merged without being reviewed and the changes properly discussed. it should be reverted as it also probably breaks in flight PRs like #684

[MarcoPolo]

related discussion: #803