test-created disposable causing side effects other tests when ran in parallel

[deeplow]

I've observed some flakiness in other tests due to the intermediate (sd-viewer-dvm-based) disposable qube combined with the introduction of parallel tests.

We may need another strategy like running that test separately or changing the test so it doesn't have side-effects.

See specific error

qube = <tests.base.QubeWrapper object at 0x739391f38ec0>
sdw_tagged_vms = <filter object at 0x739391f69990>

    def test_logs_are_flowing(qube, sdw_tagged_vms):
        """
        To test that logs work, we run a unique command in each VM we care
        about that gets logged, and then check for that string in the logs.
        """
        # Random string, to avoid collisions with other test runs
        token = "".join(secrets.choice(string.ascii_uppercase) for _ in range(10))
    
        # base template doesn't have sd-log configured
        # TODO: test a sd-viewer based dispVM
        skip = [f"sd-base-{DEBIAN_VERSION}-template", "sd-viewer"]
        # VMs we expect logs will not go to
        no_log_vms = ["sd-gpg", "sd-log"]
    
        # We first run the command in each VM, and then do a second loop to
        # look for the token in the log entry, so there's enough time for the
        # log entry to get written.
>       for vm in sdw_tagged_vms:

tests/test_vm_sd_log.py:71: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/base.py:70: in is_workstation_qube
    return is_managed_qube(qube) and SD_TAG in qube.tags
/usr/lib/python3.13/site-packages/qubesadmin/tags.py:65: in __contains__
    response = self.vm.qubesd_call(self.vm.name, 'admin.vm.tag.Get', elem)
/usr/lib/python3.13/site-packages/qubesadmin/base.py:76: in qubesd_call
    return self.app.qubesd_call(dest, method, arg, payload,
/usr/lib/python3.13/site-packages/qubesadmin/app.py:876: in qubesd_call
    return self._parse_qubesd_response(return_data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

response_data = b"2\x00QubesVMNotFoundError\x00\x00No such domain: 'sd-viewer-disposable'\x00"

    @staticmethod
    def _parse_qubesd_response(response_data):
        '''Parse response from qubesd.
    
        In case of success, return actual data. In case of error,
        raise appropriate exception.
        '''
    
        if response_data == b'':
            raise qubesadmin.exc.QubesDaemonAccessError(
                'Got empty response from qubesd. See journalctl in dom0 for '
                'details.')
    
        if response_data[0:2] == b'\x30\x00':
            return response_data[2:]
        if response_data[0:2] == b'\x32\x00':
            (_, exc_type, _traceback, format_string, args) = \
                response_data.split(b'\x00', 4)
            # drop last field because of terminating '\x00'
            args = [arg.decode() for arg in args.split(b'\x00')[:-1]]
            format_string = format_string.decode('utf-8')
            exc_type = exc_type.decode('ascii')
            try:
                exc_class = getattr(qubesadmin.exc, exc_type)
            except AttributeError:
                if exc_type.endswith('Error'):
                    exc_class = __builtins__.get(exc_type,
                        qubesadmin.exc.QubesException)
                else:
                    exc_class = qubesadmin.exc.QubesException
            # TODO: handle traceback if given
>           raise exc_class(format_string, *args)
E           qubesadmin.exc.QubesVMNotFoundError: No such domain: 'sd-viewer-disposable'

/usr/lib/python3.13/site-packages/qubesadmin/base.py:111: QubesVMNotFoundError

Originally posted by @deeplow in #1512 (comment)

Solution Proposals

Parallel run + linear run (for problematic tests)

We may need another strategy like running that test separately or changing the test so it doesn't have side-effects.

• Option 1 mark all sd-viewer tests with @pytest.mark.has_side_effects and exclude from the parallel run.

  test: test-prereqs ## Runs all application tests (no integration tests yet)
  -	pytest -v tests -v launcher/tests -n auto --dist=loadfile --durations=5
  +	# Parallel tests (no side-effects)
  +	pytest -v tests -v launcher/tests -n auto --dist=loadfile --durations=5 -m "not has_side_effects"
  +	# remaining tests
  +	pytest -v tests -v launcher/tests --durations=5 -m "has_side_effects" 

• **Option 2 ** mark like has_side_effects which we exclude from the parallel run.

  test: test-prereqs ## Runs all application tests (no integration tests yet)
  -	pytest -v tests -v launcher/tests -n auto --dist=loadfile --durations=5
  +	pytest -v tests -v launcher/tests -n auto --dist=loadfile --durations=5 --ignore tests/test_vm_sd_viewer.py

All of these approaches create a challenge because at the end we have two XML reports. It could be that OpenQA supports uploading them individually, though.

Make use of dedicated Qubes daemon and / Qubes mocks

Use Qubes' own test classes. Qubes wraps tests in their own qubesd under the hood. This makes sure tests cannot see what others tests create.

  * Can we make use of Qubes' own [qube mocking methods](https://github.com/QubesOS/qubes-core-admin-client/blob/main/qubesadmin/tests/mock_app.py) to not treat these qubes as strings, but instead have stated properties and compare with the real ones? That would be a better way to phrase tests [like these](https://github.com/freedomofpress/securedrop-workstation/blob/1.5.0/tests/test_vms_exist.py#L185-L195). For example, fedora's system upgrade tool prefixes with `system_*` and `target_*` to distinguish them.

Originally posted by @deeplow in #1339

I think this is worth exploring but has several downsides currently:

• Qubes' tests are made for nose2/unittest and not pytest (if I recall correctly)
• We'd want to provide tests with a visibility of the system state (and just obscure changes done by other tests) but this is not exacly what their tests do. They isolate the whole system, if I recall correctly.

More reading necessary.