You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: Acceptance-Tests.md
+28-46Lines changed: 28 additions & 46 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -232,66 +232,48 @@ On the first run, a screenshot named `teachPage-snap.png` will be created and st
232
232
233
233
To introduce a new screenshot to the codebase, the test should be run in all these four modes/environments to generate each screenshot in all four folders.
234
234
235
-
On CI, we run all the acceptance tests in production mode, so the screenshots in `prod-desktop-screenshots` and `prod-mobile-screenshots` will be compared to the screenshots that are generated during CI checks. If a screenshot doesn't match on CI, it generates an image in a folder as an artifact in the GitHub workflow. For example, if the screenshot `teachPage-snap.png` fails in `logged-out-user/click-all-buttons-on-navbar.spec.ts` during the CI checks in desktop environment, a folder named `diff-snapshots-logged-out-user_click-all-buttons-on-navbar` will be created. Inside this folder, a folder named `prod-desktop-screenshots` will be created and a screenshot `teachPage-diff.png` will be stored under this folder. The screenshots `teachPage-diff.png` will show the difference between the screenshot from the codebase and the new generated screenshot, making it easier to identify the difference.
235
+
On CI, we run all the acceptance tests in production mode, so the screenshots in `prod-desktop-screenshots` and `prod-mobile-screenshots` will be compared to the screenshots that are generated during CI checks. If a screenshot doesn't match on CI, it generates two images in two separate folders as artifacts in the GitHub workflow. For example, if the screenshot `teachPage-snap.png` fails in `logged-out-user/click-all-buttons-on-navbar.spec.ts` during the CI checks in desktop environment, two folders will be created, they will be named as `diff-snapshots-logged-out-user_click-all-buttons-on-navbar_desktop_original` and `new-snapshots-logged-out-user_click-all-buttons-on-navbar_desktop_original`. Inside the folder `diff-snapshots-logged-out-user_click-all-buttons-on-navbar_desktop_original`, a screenshot `teachPage-diff.png` will be generated. This screenshot `teachPage-diff.png` will show the difference between the screenshot from the codebase and the new generated screenshot, making it easier to identify the difference. And inside the folder `new-snapshots-logged-out-user_click-all-buttons-on-navbar_desktop_original`, a screenshot `teachPage-received.png` will be generated. This screenshot `teachPage-received.png` will be the new screenshot to be used to replace the old one if needed. Therefore, if we need to replace the old screenshot, we will download this screenshot from the artifacts.
236
236
237
-
On the other hand, if the screenshot fails locally (in desktop environment), the screenshot `teachPage-diff.png` will be generated and stored inside a new folder `diff-snapshots` under `logged-out-user/dev-desktop-screenshots`.
237
+
On the other hand, if the screenshot fails locally (in desktop environment), the screenshot `teachPage-diff.png` will be generated and stored inside a new folder `diff-snapshots` under `logged-out-user/dev-desktop-screenshots` and the screenshot `teachPage-received.png` will be generated and stored inside a new folder `new-snapshots` under `logged-out-user/dev-desktop-screenshots`.
238
238
239
-
### Adding Prod screenshots for Acceptance Tests
240
-
When making changes that affect a user journey tested through the acceptance tests or introduce a new feature, a contributor needs to update/add screenshots supporting their changes.
241
-
These screenshots should be obtained from the CI (not local) run, so that the environment matches future runs. To do this, you can commit a random image in the prod screenshot folder and follow these steps:
242
-
1. Go to the failing run and open the 'Uploading diff screenshots as an artifact' section:
When making changes that affect a user journey tested through the acceptance tests or introduce a new feature, a contributor needs to update screenshots to support their changes depending on where the changes are affecting. For example, if the acceptance test only fails in prod+mobile environment, then we should replace the failed screenshots in `prod-mobile-screeenshots`.
244
241
245
-
2. Then download the artifact from the artifact download url:
The screenshots in prod (`prod-desktop-screenshots` and `prod-mobile-screenshots`), should be obtained from the CI (not local) run, so that the environment matches future runs. To do this, follow these steps:
243
+
1. Go to the summary of the CI run for `full_stack_tests.yml` and scroll down to find an artifact named as `new-snapshots_{_suite_name_}_{desktop/mobile}_original`. Click on the name or the download symbol on the right hand side to download it.
The left side is the current reference image on the repo. The right side is the screenshot that was taken while running the test on the CI with your changes. The middle is the difference between the two.
252
-
253
-
We want to update the current reference as it is incorrect. Generally we do not want to do this as we expect the reference screenshots to be correct. Only do this after asking a maintainer.
246
+
2. Extract the contents of the artifact. There should be an image and the name of the image should ends with `received.png`. Rename it by replacing the `received` with `snap`. For example, rename it from `blogPage-received.png` to `blogPage-snap.png`.
254
247
255
-
4. Go to https://splitter.imageonline.co/
248
+
3. Navigate to where you have saved the Oppia repo on your local machine and go to `oppia/core/tests/puppeteer-acceptance-tests/specs`
256
249
257
-
5. Upload the diff image from the artifact you extracted:
4. Navigate to the test spec folder. The spec is mentioned in the artifact folder you downloaded in Step 1. (The spec is `blog-post-writer` in this example)
6. Go to the `prod-desktop-screenshots` or `prod-mobile-screenshots` folder depending on the failure.
265
256
266
-
8. Click on "SPLIT IMAGE".
257
+
7. Replace the screenshot having the name {screenshot_name}-snap.png with the image renamed in step 2. (Make sure to rename the pasted screenshot to {screenshot_name}-snap.png)
10. Right click on the rightmost one and use 'Save Image as...'. Choose any random name.
261
+
9. Commit and push your changes! Self review your PR to verify that the correct image(s) were used.
272
262
273
-
11. Navigate to where you have saved the Oppia repo on your local machine and go to oppia/core/tests/puppeteer-acceptance-tests/specs
263
+
For the screenshots in dev (`dev-desktop-screenshots` and `dev-mobile-screenshots`), follow these steps to update the screenshots:
264
+
1. Navigate to where you have saved the Oppia repo on your local machine and go to `oppia/core/tests/puppeteer-acceptance-tests/specs`
274
265
275
-
12. The spec will be mentioned in the artifact folder you downloaded in Step 2:
266
+
2. Navigate to the test spec folder. For example, the spec would be `blog-post-writer` if the screenshot is under the spec `blog-post-writer/create-and-edit-blog-post`.
276
267
277
-
<imgwidth="368"height="64"alt="Screenshot from 2025-08-20 16-37-52"src="https://github.com/user-attachments/assets/d1a45271-c1f1-49d9-8874-f5ac95d37dce" />
268
+
3. Go to the `dev-desktop-screenshots`or `dev-mobile-screenshots` folder depending on the failure.
278
269
279
-
(The spec is curriculum-admin in this case)
270
+
4. Navigate to `new-snapshot` and rename the image in the folder by replacing the `received` with `snap`.
280
271
281
-
13. Navigate to the test spec folder based on the above step.
272
+
5. Navigate back to the `dev-desktop-screenshots`or `dev-mobile-screenshots` folder depending on the failure. Replace the screenshot having the name {screenshot_name}-snap.png with the image renamed in step 4. (Make sure to rename the pasted screenshot to {screenshot_name}-snap.png)
282
273
283
-
14. Check if the failure was for desktop or mobile based on the screenshot or the subfolder in the artifact:
6. Check that the correct image got replaced. Run the test locally to check if the test passes.
285
275
286
-
15. Go to the prod-desktop-screenshots or prod-mobile-screenshots folder depending on your failure.
287
-
288
-
16. Check the name of the image from the artifact. It will be {screenshot_name}-diff.png
289
-
290
-
17. Replace the screenshot having the name {screenshot_name}-snap.png with the split image we saved in step 10. (Make sure to rename the pasted screenshot to {screenshot_name}-snap.png)
291
-
292
-
18. Check that the correct image got replaced.
293
-
294
-
19. Commit and push your changes! Self review your PR to verify that the correct image(s) were used.
276
+
7. Commit and push your changes! Self review your PR to verify that the correct image(s) were used.
295
277
296
278
## Acceptance Tests for Mobile
297
279
@@ -305,7 +287,7 @@ However, in scenarios where certain actions are affected by the smaller screen s
305
287
306
288
For example: consider a scenario where a menu is collapsed into a hamburger menu due to the small screen size:
Here, if we want to click on the "Home" or any other button, we need to first click on the hamburger menu. Additionally, there may be differences in selectors for the same buttons between desktop and mobile. For instance, the publish button in desktop might be `e2e-test-publish-exploration`, while in mobile it could be `e2e-test-mobile-publish-button`.
311
293
@@ -419,11 +401,11 @@ To address this:
419
401
* Trigger the workflow manually in your fork using the following steps:
420
402
1. Navigate to your fork (github.com/YOUR_USERNAME/oppia).
421
403
2. Sync your fork with the upstream repository (oppia/oppia).
422
-
1. Click on "Sync Fork" button in the top right corner of the fork page. 
423
-
2. Click on "Update Branch" button. 
404
+
1. Click on "Sync Fork" button in the top right corner of the fork page. 
405
+
2. Click on "Update Branch" button. 
424
406
3. Navigate to the Actions tab in top menu. Then, click on the workflow "Stress Test Acceptance Tests" from the newly opened left menu.
0 commit comments