Rearchitect test framework

[mattmcneeney]

As things stand, the entire suite of tests will have to be duplicated for each minor version (2.13, 2.14, 2.15, etc). However, as these minor versions must be backwards compatible, new minor versions should just mean adding new tests to the suite.

When v3 eventually lands, we'd likely need a brand new suite of tests (similar to how things work today for adding minor versions).

Is anyone up for having a go at this?

[jeremyrickard]

@mattmcneeney I'd be interested in having a go at this.

[mattmcneeney]

Awesome, thanks @jeremyrickard.

@n3wscott @Haishi2016 @arschles Any thoughts on this proposal before this gets underway?

[n3wscott]

👍 do it! maybe the test can know how it matches using semver and path matching? like 2.* and 2.13+ and *, etc...

[arschles]

I'm good with this proposal. It also makes it easier (almost free) to have testing for backward compatibility if we just add tests.

I second the matching logic that @n3wscott recommends for 2.* versions as well. I think we can use that while still getting the backward compatibility tests "for free".

For 3.* versions, I don't think we can predict what those tests will look like because we can't predict what the spec will look like, but I do think we can at least take the semver matching logic and use it in the 3.x tests. Possibly the path matching logic too

[mattmcneeney]

Hey @jeremyrickard - do you still have time to look into the re-architecture of this framework soon?

[jeremyrickard]

@mattmcneeney I started looking at this but then got caught up doing some higher priority things, so I haven't really made much progress on it.

[leonwanghui]

Please put me in if we start the design, thx!

[leonwanghui]

@norshtein @zhongyi-zhang Any thought on this issue? I think semver is a good suggestion.