Flaky - Areas of screenshots blanked out `browsingContext.captureScreenshot`

[Venefilyn]

Over on Cockpit we're heavy users of visual testing and uses the bidi drivers for Chromium and Firefox with out own visual testing implementation. We first noticed this happening with buttons/sections being completely gone from screenshots back in February, then it developed to sections within screenshots being blanked out/white when we migrated from Patternfly v5 to v6.

Let us know if there is anything we can do to help 🙏

Reproducible

There is no reproducible way to get these bugs but we run tests often enough that we do get quite a bit of errors on our PRs, main branch, or daily test runs.

Setup

Our bidi driver for Chromium is setup using headless (Edit: Chromium binary with --headless), full configuration:
https://github.com/cockpit-project/cockpit/blob/c2a96d076f9d285718c4613a983e7ac15b8c9945/test/common/webdriver_bidi.py#L360-L381

Workaround

We have started doing a workaround where any tests we discover to have flakyness related to rendering sections as blank we render it again just to be sure it isn't a flake. In most cases this has fixed the issue but it keeps popping up in other tests and we'd rather get this fixed at the root.

Code for our workaround
https://github.com/cockpit-project/cockpit/blob/db792bcd96e8942fef0a34d249ea5eb9c334ac5d/test/common/testlib.py#L1370-L1392

Test examples

Issues with CI pixel tests and raw screenshot

1. check-application#TestFiles.testPermissions - PR
2. check-metrics#TestHistoryMetrics-testEvents (screenshot) - PR - Temporary pixel test comparison
3. check-shell-keys#TestKeys.testPrivateKeys - (screenshot) - PR - Temporary pixel test comparison
4. check-superuser#TestSuperuser.testBasic - (screenshot) - PR

Other screenshots

blue 'Change' button missing next to 'Cancel'

blue 'Change' button missing next to 'Cancel'

Issues/PRs from Cockpit

• Buttons or performance profile randomly disappear cockpit-project/cockpit#21577

[Venefilyn]

@sadym-chromium I have this and #3421. Want me to submit this in a different tracker or is this fine? Just wanted to make sure so this doesn't fall between the cracks 🙏

[Venefilyn]

@sadym-chromium hate to ping again but should I submit this and #3421 somewhere else?

[sadym-chromium]

looking in

[Venefilyn]

looking in

Thanks @sadym-chromium!

---

We just got the issue again here.

Rendering should look like this. In the top right you can see Serial and Graphical buttons being visible as expected

But with the bug it sometimes becomes like this. Now in the top right the Serial and Graphical buttons are missing. They exist in the DOM and everything is there and the best way to fix it is to call captureScreenshot again