Support both pytest and legacy system test runner

The legacy system test framework uses pytest to execute some tests.
Since it'd be quite difficult to convince pytest to decide whether to
include conftest.py (or which ones to include when launching from
subdir), it makes more sense to have a shared conftest.py which is used
by both the legacy test runner invocations of pytest and the new pytest
system test runner. It is ugly, but once we drop support for the legacy
runner, we'll get rid of it.

Properly scope the *port fixtures in order to ensure they'll work as
expected with the new pytest runner. Instead of using "session" (which
means the fixture is only evaluated once for the entire execution of
pytest), use "module" scope, which is evaluated separately for each
module. The legacy runner invoked pytest for each system test
separately, while the new pytest runner is invoked once for all system
tests -- therefore it requires the more fine-grained "module" scope to
for the fixtures to work properly.

Remove python shebang, as conftest.py isn't supposed to be an executable
script.

Author: nicki-krizek

bin/tests/system/conftest.py

@@ -1,5 +1,3 @@
-#!/usr/bin/python3
-
 # Copyright (C) Internet Systems Consortium, Inc. ("ISC")
 #
 # SPDX-License-Identifier: MPL-2.0
@@ -12,25 +10,43 @@
 # information regarding copyright ownership.
 
 import os
-
 import pytest
 
 
-@pytest.fixture(scope="session")
+# ======================= LEGACY=COMPATIBLE FIXTURES =========================
+# The following fixtures are designed to work with both pytest system test
+# runner and the legacy system test framework.
+
+
+@pytest.fixture(scope="module")
 def named_port():
     return int(os.environ.get("PORT", default=5300))
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="module")
 def named_tlsport():
     return int(os.environ.get("TLSPORT", default=8853))
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="module")
 def named_httpsport():
     return int(os.environ.get("HTTPSPORT", default=4443))
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="module")
 def control_port():
     return int(os.environ.get("CONTROLPORT", default=9953))
+
+
+# ======================= PYTEST SYSTEM TEST RUNNER ==========================
+# From this point onward, any setting, fixtures or functions only apply to the
+# new pytest runner. Ideally, these would be in a separate file. However, due
+# to how pytest works and how it's used by the legacy runner, the best approach
+# is to have everything in this file to avoid duplication and set the
+# LEGACY_TEST_RUNNER if pytest is executed from the legacy framework.
+#
+# FUTURE: Once legacy runner is no longer supported, remove the env var and
+# don't branch the code.
+
+if os.getenv("LEGACY_TEST_RUNNER", "0") == "0":
+    pass  # will be implemented in followup commits

bin/tests/system/run.sh.in

@@ -229,7 +229,7 @@ if [ $status -eq 0 ]; then
             if start_servers; then
                 run=$((run+1))
                 test_status=0
-                (cd "$systest" && "$PYTEST" -rsxX -v "$test" "$@" || echo "$?" > "$test.status") | SYSTESTDIR="$systest" cat_d
+                (cd "$systest" && LEGACY_TEST_RUNNER=1 "$PYTEST" -rsxX -v "$test" "$@" || echo "$?" > "$test.status") | SYSTESTDIR="$systest" cat_d
                 if [ -f "$systest/$test.status" ]; then
                     if [ "$(cat "$systest/$test.status")" = "5" ]; then
                         echowarn "R:$systest:SKIPPED"