Refs #34043 -- Added GitHub action to capture screenshots in Selenium tests.

[sarahboyce]

This is a proposal to help us confidently make UI changes. 💪
Testing UI is always hard and many of us would not describe ourselves as frontend developers. Adding a layer of visual testing to our automated test suite will not only make our lives easier but can open up possibilities to us such as tests to cover visual regressions.

In ticket 34043, there is a suggestion that we should have a test suite/tooling to keep track of UI changes automatically.

This PR adds the screenshot tests to our GitHub actions. This is currently triggered when a PR has the selenium label.
You can see the screenshots (in a zip file at the bottom called screenshots) here: https://github.com/django/django/actions/runs/5386413722?pr=16963

The action "fails" because I do not have write permissions to the repo. please see this comment #16963 (comment). I have tested this against my fork where I do have write permissions. This should be resolved on merge.

[smithdc1]

I'm not sure if this is helpful? Maybe you've already seen it!

https://simonwillison.net/2022/Mar/10/shot-scraper/

[sarahboyce]

I'm not sure if this is helpful? Maybe you've already seen it!

https://simonwillison.net/2022/Mar/10/shot-scraper/

I haven't but it's quite similar (I am also using playwright here but using the python version because Python = ❤️), will take a look!
I am trying also take inspiration from Adam's blog (https://adamj.eu/tech/2023/03/17/django-parameterized-tests-model-admin-classes/) around testing the admin and what I want in the end is something that will give you just changed screenshots with before and after so hopefully the css changes are easier when seeing unexpected changes (visual regressions are just difficult)

As an example, you can download and see screenshots here in the Artifacts section at the bottom: https://github.com/django/django/actions/runs/5225589580, it's not quite what I want yet but you get the idea. Need to check with people if we think this adds value.

Crossing out comment as the approach has since changed, leaving this comment misleading.

[felixxm]

@sarahboyce Thanks for all your efforts 👍 I would like to merge/discuss the remaining code in two steps:

1. new screenshots option to the runtests.py,
2. new GitHub action.

[felixxm]

I moved the first commit with small edits (release notes, comments) to the #17374.

[felixxm]

I moved the first commit with small edits (release notes, comments) to the #17374.

Merged in be56c98.

[felixxm]

@sarahboyce Many thanks! rebased.

[felixxm]

I simplified the generated comment a bit to:

[ngnpope]

I simplified the generated comment a bit to:

This reverts a bunch of what was discussed in this comment, in particular reducing the visibility of screenshots taken on previous runs. Being able to easily access multiple downloads is useful for comparison.

Maybe an alternative is not to edit existing comments, but just create new ones every time?

[felixxm]

Maybe an alternative is not to edit existing comments, but just create new ones every time?

Yes, I was thinking about the same 👍

[felixxm]

@ngnpope You can check the result in felixxm#6.

[felixxm]

This also make it easier to compare results when pushing fixes.

[sarahboyce]

Maybe an alternative is not to edit existing comments, but just create new ones every time?

Yes, I was thinking about the same 👍

I'm not a fan of a new message with each push/commit. When you get enough comments, like this PR, conversations become hidden even when they're not resolved. I missed that Nick's suggestion earlier was to append to an existing message. Can try that? Or that a message is created in a thread? Worried a new message each push/commit might pollute a review process 🤔

[felixxm]

Let's check how it works with updating a comment, felixxm#8.

[felixxm]

All good, but date is not there 🤔

[felixxm]

Finally, it works! 🎉 felixxm#9

[sarahboyce]

Finally, it works! 🎉 felixxm#9

Ah I love it!