Modularize test suite

[pederhan]

The test suite is growing quite large, and for each new test we add, cleaning up created/modified resources becomes more and more important so it doesn't affect the subsequent tests. All this setup/teardown, as well as just the growing number of commands being tested, is bloating both the test file and test results file.

Current implementation

Right now, commands are loosely grouped by category inside the test suite file:

mreg-cli/ci/testsuite

Lines 1 to 14 in bc04268

	##### Tests: Zone
	# - zone creation
	# - zone info
	# - zone list
	# - set/update ns, soa, ttl
	zone create example.org hostmaster@example.org ns1.example.org # should require force because ns1 is unknown
	zone create example.org hostmaster@example.org ns1.example.org -force
	zone info example.org |! Serialnumber # Serialnumber includes the date: https://github.com/unioslo/mreg/blob/master/mreg/utils.py#L78
	zone list -forward
	zone set_ns example.org ns2.example.org # requires force because ns2 is unknown
	zone set_ns example.org ns2.example.org -force
	zone set_soa example.org -email hostperson@example.org -serialno 12345 -refresh 360 -retry 1800 -expire 2400 -soa-ttl 1800
	zone set_default_ttl example.org 60 # should fail, must be >= 300
	zone set_default_ttl example.org 300

mreg-cli/ci/testsuite

Lines 17 to 39 in bc04268

	##### Tests: Network
	# - list used/unused IPv4 addresses
	# - pick next unused address
	# - set number of reserved addresses
	network create -network 10.0.2.0/28 -desc "TinyNet"
	network list_used_addresses 10.0.2.0/28
	network list_unused_addresses 10.0.2.0/28
	network set_reserved 10.0.2.0/28 8
	host add tinyhost -ip 10.0.2.0/28 -contact tinyhost@example.org
	network list_used_addresses 10.0.2.0/28
	network list_unused_addresses 10.0.2.0/28
	host remove tinyhost
	network remove 10.0.2.0/28 -force
	# ipv6
	network create -network 2001:db8::/64 -desc "Lorem ipsum dolor sit amet"
	network list_used_addresses 2001:db8::/64
	network list_unused_addresses 2001:db8::/64
	network set_reserved 2001:db8::/64 50
	host add tinyhost -ip 2001:db8::/64 -contact me@example.org
	host info tinyhost | 2001
	host info tinyhost |! example.org
	host remove tinyhost
	network remove 2001:db8::/64 -f

etc.

This works OK, but it can be hard to navigate and has all the aforementioned problems. It is also unclear if (and if so which) commands have downstream effects on other tests.

Breaking it up

I propose we modularize the test suite into units that can test specific categories of commands. This should make it easier to add/remove tests without causing a cascade of changes to later tests. Each category of commands is stored in a separate file in a new commands directory, with their corresponding result files in results.

.
└── testsuite
    ├── commands
    │   ├── _common
    │   ├── host
    │   ├── hostpolicy
    │   ├── network
    │   └── zone
    └── results
        ├── new
        │   ├── host.json
        │   ├── hostpolicy.json
        │   ├── network.json
        │   └── zone.json
        ├── host.json
        ├── hostpolicy.json
        ├── network.json
        └── zone.json

Common commands

The ascii tree above shows a _common file, which can be used to execute a set of commands common to all tests. This would primarily be used to set up the zone(s) for all tests. We could run this in a separate mreg-cli invocation before the actual tests to avoid adding it to the test results:

mreg-cli --source commands/_common
mreg-cli --source commands/host --record results/new/host.json

Technical implementation

Adapt script to iterate over test files

Pseudocode:

# Run all test suite files first
for commandfile in Path("commands"):
    run_common()
    run(f"mreg-cli --source {commandfile} --record results/new/{commandfile}.json")
    wipe_database()

# Diff all result files after
for resultfile in Path("results"):
    run(f"diff.py {resultfile} new/{resultfile}")

Not shown in the example is some way to correlate command files with result files. Ideally, we populate some sort of data structure with this before iterating, so we have a single data structure that produces the expected file paths:

class Commands:
    commands: Path

    def __init__(self, commands: Path) -> None:
        self.commands = commands

    @property
    def name(self) -> str:
        """Get the name of the command category."""
        return self.commands.name

    @property
    def basedir(self) -> Path:
        """Get the path to the base directory for the test suite."""
        return self.commands.parent.parent

    @property
    def results_dir(self) -> Path:
        """Get the path to the results directory."""
        return self.basedir / "results"

    @property
    def results(self) -> Path:
        """Get the path to the base results file."""
        return self.results_dir / self.commands.with_suffix(".json").name

    @property
    def new_results(self) -> Path:
        """Get the path to the new results file."""
        return self.results_dir / "new" / self.commands.with_suffix(".json").name

Wiping the database

In order to not have tests affect each other, we need to either wipe the MREG database between tests or spin up a new container for category. A database wipe is certainly the fastest.

Perhaps all we need is:

uv run manage.py flush