Add SecureDrop installation test #25

deeplow · 2024-09-11T13:11:42Z

First attempt at adding a test for SecureDrop.

marmarek · 2024-09-11T14:04:00Z

tests/securedrop/install.pm

+    assert_and_click("menu-vm-xterm");
+
+
+    assert_script_run('gpg --keyserver hkps://keys.openpgp.org --recv-key "2359 E653 8C06 13E6 5295 5E6C 188E DD3B 7B22 E6A3"');


assert_script_run depends on seeing serial console output - serial console from "work" VM isn't directly connected to the one of the host; for this to work you either need to run something like tail -F /var/log/xen/console/guest-work.log >> /dev/hvc0 in dom0 (we do that here), or do all that from dom0's terminal via qvm-run

I see. Would type_string and then "ret" work as well? I'm trying not to deviate to much from the original instructions so it's easy to update.

Yes, that would work, but your test wouldn't detect if any of those command fails (other than possible some later step dom0 in dom0 failing).

Fair point 😔. I'll just go ahead an use qvm-run, then.

tests/securedrop/install.pm

marmarek · 2024-09-12T11:03:12Z

Hint: add send_key('alt-f10') to see more output at once in xterm. Not relevant much when everything goes right, but helps quite a bit when debugging.

deeplow · 2024-09-12T12:56:00Z

Hint: add send_key('alt-f10') to see more output at once in xterm. Not relevant much when everything goes right, but helps quite a bit when debugging.

Thanks for the tip. I had seen that in some places and was wondering about its purpose. I'll add it in the next round.

deeplow · 2024-09-12T18:25:18Z

tests/securedrop/install.pm

+    assert_script_run('curl https://raw.githubusercontent.com/freedomofpress/securedrop/d91dc67/securedrop/tests/files/test_journalist_key.sec.no_passphrase | sudo tee /usr/share/securedrop-workstation-dom0-config/sd-journalist.sec');
+    assert_script_run('sdw-admin --validate');
+
+    assert_script_run('xfce4-power-manager -q'); # disable screen blanking during long command


@marmarek there's a command which takes quite a while and in the meantime the screen blanks. I don't think it's xscreensaver because I think that's killed at the beginning of the test. Then I tried to disable XFCE's power management, but didn't help.

Have you encountered this before?

My notes have this line:

x11_start_program('env xset s off', valid => 0);

but I'm not sure if that was enough either.

Thanks! I had to combine it with env xset -dpms for this to fully work.

And FYI I noticed that just with env xset s off it still blanked for a lot of the slow command (sdw-admin --apply), but oddly enough the screen showed up just the logs upload command (video). No idea what went on there.

It unblanked on the key press.

Oh! I totally forgot that it was literally typing each letter. That's why, then.

I recall that the above options were still not working perfectly (the screeen was still bllanking at some point). What seems to have solved it is enabling presentation mode. I haven't look at what it's doing under the hood. But it seems to work. And because the setting is persistent, I think it shouldn't need anymore all the xscreensaver exits.

marmarek · 2024-09-12T19:43:13Z

Anyway:

# Test died: command 'sdw-admin --apply' timed out at /usr/lib/os-autoinst/autotest.pm line 412.

So, longer timeout? This is running virtualized, so runs slower than native.

And also, I recommend collecting and uploading logs. For example wrap it with script, or use tee (see https://github.com/QubesOS/openqa-tests-qubesos/blob/main/tests/update2.pm#L110-L111 for example). You can also do post-fail hook to collect extra info on failure too: https://github.com/QubesOS/openqa-tests-qubesos/blob/main/tests/update2.pm#L168-L174

deeplow · 2024-09-16T14:03:32Z

So, longer timeout? This is running virtualized, so runs slower than native.

Fair point. I have added some timeout.

Now I am running into another issue. I have created a needle through the web interface added for this step an assert_and_click. However, when it runs, it's not even listing the needle. Do I need to add the needle's PNG and respective JSON to the commit?

marmarek · 2024-09-16T14:21:35Z

Have you restarted the test after adding the needle? Or did you added it via developer mode?

deeplow · 2024-09-16T15:07:36Z

I thought I had restarted it afterwards. But will try again. It for sure wasn't via developer mode. Let's see if it now finds the needle.

marmarek · 2024-09-16T15:48:14Z

I see the issue: you haven't added the securedrop-launcher tag, it only has desktop tag (which shouldn't be there I think). I guess you added it by clicking on an earlier screenshot (you can do that too, but then you need to adjust tags manually, as the default will be about that other screenshot).

deeplow · 2024-09-16T15:52:06Z

OK. Makes sense. I was afraid to create new tags. Where can I edit the needle? Or should I create a new one?

marmarek · 2024-09-16T15:57:58Z

For this one I just edited it manually.
But generally create new one, and don't be afraid about adding tags. In fact, do add more of them :) for SD-specific needles add ENV-securedrop tag (in addition to any others).

marmarek · 2025-06-12T08:26:13Z

Check now (without changes on your side)
I disabled auto-update, I think default for this option changed recently.

deeplow · 2025-06-12T08:59:34Z

Sadly it seems to have failed. I'm still waiting on the one with NEEDLES_DIR stated explicitly (https://openqa.qubes-os.org/tests/142851).

deeplow · 2025-06-12T10:47:41Z

The NEEDLES_DIR one is running just fine. https://openqa.qubes-os.org/tests/142851#settings

marmarek · 2025-08-31T10:38:34Z

@deeplow sorry, I might have broken SD tests by enabling presentation mode in the base image already...

deeplow · 2025-09-01T15:00:20Z

No problem at all. If it's in the base image already, then it's even better. I think quickly noticed and worked around it in the only test where it broke at the time. But I haven't seen other failures even without the workaround.

In any case I have a few things it the oven for securedrop's OpenQA tests and I'll remove that bit in case it's already in the base image.

deeplow · 2025-09-01T15:10:22Z

@marmarek Speaking of which, if you have a chance, could help me set up a base image for SecureDrop? Originally we were just thinking of pre-downloading debian-12-minimal, but there are a few other things that could help (pre-setup the securedrop server), run whonix updates, etc.

I'm guessing implementation-wise this could be with the SECUREDROP_PREP variable, to be built on top of the base image. In terms of build cadence, I think having this run with the same frequency as the base image is, should be fine, but it could be that even a lower frequency works as well.

Is there anything else that would be needed from our end to make this work? (other than afterwards opening a PR with the respective main.pm changes)

Now done in base test image. Per discussion in QubesOS#25 (comment)

marmarek · 2025-09-01T17:14:29Z

There are two options:

standard base image job (part of weekly builds) -> SD preparation job (creates "SD base image") - the latter would run either after weekly builds, or on some other schedule
separate base image job, with SD-specific preparation included already - running in parallel to other weekly jobs

The second one may not be very time-effective, as it's doing installation again, but it's done on a schedule (doesn't delay PR results), and have some benefits: you can adjust installation itself - for example don't install templates you don't need.

BTW, part of what you need is already implemented - for example you can set INSTALL_TEMPLATES=debian-12-minimal to install extra templates. I try to document supported variables in README, but I'm sure some are missing there...
In fact, there is already a job that adds debian-12-minimal template too (and few others), see https://openqa.qubes-os.org/tests/151215

Both approaches would need some variable (can be SECUREDROP_PREP) and relevant main.pm snippet in the same place. The difference is only which other variables will be set in that job (especially - if ISO=... for fresh install, or HDD_1 for extending existing base image).

deeplow · 2025-09-01T17:55:21Z

It seems that option 1. could be more suitable. I think we'll want to build on top of what already exists as long as we can have this "SD base image" job already done in advance in a way that doesn't delay the PRs (which both approaches seem to do).

BTW, part of what you need is already implemented - for example you can set INSTALL_TEMPLATES=debian-12-minimal to install extra templates. I try to document supported variables in README, but I'm sure some are missing there...

That's great to know. Although from the looks of it, it seems that this would only work if we went down approach 2 as it's part of the anaconda tests. But I think it's fine and we also get more flexibility by doing it in a perl file, and all the "prep"-related code is all in the same place place.

marmarek · 2025-09-01T17:59:00Z

Although from the looks of it, it seems that this would only work if we went down approach 2

Not really, in the job I linked it's used this way, but this feature can be used in other jobs too (then, the setting is named UPDATE_TEMPLATES, but the behavior is the same).

marmarek · 2025-10-04T02:38:32Z

@deeplow recent failure has this in xen log:

(XEN) d14v0 Triple fault - invoking HVM shutdown action 1
(XEN) *** Dumping Dom14 vcpu#0 state: ***
(XEN) ----[ Xen-4.17.5  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    0008:[<00000000000f5eee>]
(XEN) RFLAGS: 0000000000010006   CONTEXT: hvm guest (d14v0)
(XEN) rax: 0000000000000060   rbx: 000000002030e623   rcx: 0000000085fffffe
(XEN) rdx: 00000000188e7fd6   rsi: 0000000000000000   rdi: 00000000008dc878
(XEN) rbp: 0000000000000000   rsp: 00000000188e7fec   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000000000011   cr4: 0000000000000000
(XEN) cr3: 0000000000000000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) ds: 0010   es: 0010   fs: 0010   gs: 0010   ss: 0010   cs: 0008

But IIUC that test was from the main branch of securedrop-workstation repo, not a PR. I don't think there were any recent changes in Xen, so maybe some change in the VM kernel? Which one is used there (for sys-net)? Is it maybe a grsec one (I think it isn't but just to be sure)? Recent test run of R4.2 updates (no SD) with default kernel (6.12.47) was okay...

deeplow · 2025-10-06T09:49:52Z

But IIUC that test was from the main branch of securedrop-workstation repo, not a PR. I don't think there were any recent changes in Xen, so maybe some change in the VM kernel? Which one is used there (for sys-net)?

I had not seen this one yet. sys-net has not custom modifications, the only thing we're doing as far as sys-net is concerned is updating the base fedora template to fedora-42-xfce if it wasn't already the case. All modifications should be contained here) and from what I can see in the video, it failed after change change the base template. (sadly the post-fail hook doesn't seem to have uploaded the salt log).

So my assumption is that this is an issue that would have happened in "vanilla" Qubes with a Fedora 42 as the sys-net template.

deeplow · 2025-10-07T12:38:49Z

@marmarek There was another instance of sys-net failing just yesterday (see here), but I don't find a similar error message. I looked in hypervisor.log and in the guest-sys-net.log.

marmarek · 2025-10-07T13:18:03Z

but there is

[2025-10-06 14:15:01] (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:03.0] fault addr fe1457b4fe145000
[2025-10-06 14:15:01] (XEN) [VT-D]DMAR: reason 04 - Access beyond MGAW

and the other one also has those, but a bit earlier

What is most interesting here, is that sys-net (when it starts correctly) doesn't actually have 0000:00:03.0 device at all...

deeplow · 2025-10-25T12:40:04Z

I've started a test with 1080p as resolution here, since you had suggested that it was now possible. To do that I have set the OpenQA XRES and YRES. Everything appears to be working correctly but I wanted to check with you that I was doing it properly.

Should we have any extra concerns? (I assume video size will increase, but among all other storage, this is probably not the biggest concern).

marmarek · 2025-10-25T13:53:31Z

For tests on VMs that's enough. For tests on real hw there would need to be some more settings (especially - EDID with matching resolution).

deeplow · 2025-10-25T13:55:34Z

Perfect! Thanks!

deeplow · 2025-12-10T10:17:31Z

@marmarek this is the line I mentioned yesterday that I had to uncomment to also have qubes-dist-upgrade installed from the repos instead of fetching from a fork. Just FYI.

marmarek · 2025-12-10T11:15:04Z

491ed9f

deeplow · 2025-12-10T11:55:05Z

I should have looked at the real repo. Thanks!

The NEEDLES_DIR was a temporary override due to an issue in OpenQA [1], where tests would fail with the following message: Gru job failed Reason: NOT updating dirty Git checkout at '/var/lib/openqa/share/tests/qubesos/needles'. In case this is expected (e.g. on a development openQA instance) you can disable auto-updating. Then the Git checkout will no longer be kept up-to-date, though. Checkout http://open.qa/docs/#_getting_tests for details. This seems to have been resolved in the meantime, so it's time to re-enable it. Another reason to do so is that with GUI tests we want to be able to live-edit needles directly on OpenQA, interactively. But if we override that directly it only knows about the ones in the repo and not the ones we just created and therefore, it fails to find the needle. Since at the time we had not yet actively worked on GUI test (which need needles, we had not yet run into problems with our temporary workaround) [1]: QubesOS/openqa-tests-qubesos#25 (comment)

deeplow · 2026-01-07T11:17:05Z

FYI, I've noticed some failures that don't seem to be SecureDrop-related:

It doesn't always happen, but I think for the time being we can work around it. And it doesn't seem to be OpenQA worker-related, this time.

marmarek · 2026-01-07T11:43:02Z

Ramen has some slower download speed for some reason, you can hit restart and it should be good. The actual download continues in the background, just test stopped waiting for it.

deeplow · 2026-01-07T17:03:52Z

OK. Thanks.

marmarek · 2026-02-03T19:05:24Z

@deeplow I see all 4.3 tests are failing on missing debian-12-xfce template. Do you need some help getting it installed? AFAIR the plan is to stick to 12 for now, right?

deeplow · 2026-02-03T19:13:46Z

Thanks! I think I have it sorted out in https://github.com/freedomofpress/openqa-tests-qubesos/pull/10/files. Mostly just waiting on review from someone on the SD team. But it won't pass until we also have freedomofpress/securedrop-workstation#1373 reviewed.

But I might mention that I ran into an issue where preloaded disposables were not enabled by default on tests, so enabled them (see linked PR). Am I correct in the expectation that they are disabled by default? And if so is there a better way to enable in OpenQA than running the salt states that the installer uses?

marmarek · 2026-02-03T19:21:24Z

Am I correct in the expectation that they are disabled by default?

They are enabled based on available memory (at install time). Tests (virtualized) are running with 8GB, which doesn't qualify for enabling them by default. We have a test where they are enabled anyway, by setting DISPVM_PRELOAD=1 in job settings, but note it happens after SD tests in main.pm (you might want to move it).

deeplow · 2026-02-03T19:27:54Z

They are enabled based on available memory (at install time).

Makes sense. That was my working assumption.

by setting DISPVM_PRELOAD=1 in job settings, but note it happens after SD tests in main.pm (you might want to move it).

I'll do that then. Thanks!

marmarek reviewed Sep 11, 2024

View reviewed changes

tests/securedrop/install.pm Outdated Show resolved Hide resolved

deeplow force-pushed the add-securedrop-test branch 2 times, most recently from 3502fd7 to 2008e3f Compare September 12, 2024 10:54

deeplow force-pushed the add-securedrop-test branch 6 times, most recently from 5c8b79c to 3a2149a Compare September 12, 2024 12:39

deeplow force-pushed the add-securedrop-test branch from 3a2149a to deebce7 Compare September 12, 2024 17:29

deeplow commented Sep 12, 2024

View reviewed changes

deeplow force-pushed the add-securedrop-test branch 8 times, most recently from bf7f90e to fb294a2 Compare September 13, 2024 16:48

deeplow force-pushed the add-securedrop-test branch from fb294a2 to ff78699 Compare September 16, 2024 18:35

deeplow added a commit to deeplow/openqa-tests-qubesos that referenced this pull request Sep 1, 2025

Remove presentation mode enabling

235a5d9

Now done in base test image. Per discussion in QubesOS#25 (comment)

deeplow mentioned this pull request Dec 17, 2025

Unbreak OpenQA tests with new parameters freedomofpress/securedrop-workstation#1524

Merged

2 tasks

		assert_and_click("menu-vm-xterm");


		assert_script_run('gpg --keyserver hkps://keys.openpgp.org --recv-key "2359 E653 8C06 13E6 5295 5E6C 188E DD3B 7B22 E6A3"');

Uh oh!

Add SecureDrop installation test #25

Add SecureDrop installation test #25

Uh oh!

Conversation

deeplow commented Sep 11, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

marmarek commented Sep 12, 2024

Uh oh!

deeplow commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deeplow Oct 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marmarek commented Sep 12, 2024

Uh oh!

deeplow commented Sep 16, 2024

Uh oh!

marmarek commented Sep 16, 2024

Uh oh!

deeplow commented Sep 16, 2024

Uh oh!

marmarek commented Sep 16, 2024

Uh oh!

deeplow commented Sep 16, 2024

Uh oh!

marmarek commented Sep 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marmarek commented Jun 12, 2025

Uh oh!

deeplow commented Jun 12, 2025

Uh oh!

deeplow commented Jun 12, 2025

Uh oh!

marmarek commented Aug 31, 2025

Uh oh!

deeplow commented Sep 1, 2025

Uh oh!

deeplow commented Sep 1, 2025

Uh oh!

marmarek commented Sep 1, 2025

Uh oh!

deeplow commented Sep 1, 2025

Uh oh!

marmarek commented Sep 1, 2025

Uh oh!

marmarek commented Oct 4, 2025

Uh oh!

deeplow commented Oct 6, 2025

Uh oh!

deeplow commented Oct 7, 2025

Uh oh!

marmarek commented Oct 7, 2025

Uh oh!

deeplow commented Oct 25, 2025

Uh oh!

marmarek commented Oct 25, 2025

Uh oh!

deeplow commented Sep 12, 2024 •

edited

Loading

deeplow Oct 11, 2024 •

edited

Loading

marmarek commented Sep 16, 2024 •

edited

Loading

deeplow commented Dec 10, 2025 •

edited

Loading

deeplow commented Feb 3, 2026 •

edited

Loading

marmarek commented Feb 3, 2026 •

edited

Loading