Automated tests for api version upgrades

[jmcarp]

We've been testing that the provider behaves reasonably when upgrading nexus/omicron versions, but AIUI this is a fairly annoying process. Either we work through the upgrade on a test rack, or do state surgery to simulate an upgrade. I'd like to get this process working entirely within simulated omicron if possible, which I think we can do using the "second nexus" pattern: run cargo xtask omicron-dev run-all on the old omicron version, then run a second nexus instance on the new omicron version.

A small complication: run-all uses random ports for all services, so the current process requires collecting the randomly selected ports from the log output and pasting them into a config file for the auxiliary nexus instance. I'm proposing a small change in oxidecomputer/omicron#9696 that allows us to choose the relevant ports, which should make this easier to automate without parsing log output.

cc @lgfa29

[sudomateo]

From reading this, it seems the goal is to have some test that creates resources on Oxide version N-1, upgrades Oxide to version N, and runs Terraform again on Oxide version N to verify a no-op plan. Is that correct or am I missing something?

I understand the acceptance tests clean up resources after then run so running them against N-1 and N doesn't give us the upgrade signal we're looking for. To this point, do we plan on using the acceptance testing harness to perform this upgrade logic? Just trying to get a higher level understanding.