[Issue #8184] allow usage of playwright tags in e2e test jobs by doug-s-nava · Pull Request #8960 · HHS/simpler-grants-gov

doug-s-nava · 2026-03-10T19:18:31Z

Summary

Fixes for #8184

Changes proposed

Adds support for limiting which playwright tests are run in our e2e testing ci jobs by passing playwright tags.

Context for reviewers

Includes definition of groups to two tests - these are meant to facilitate verification and manual testing of this PR, but the assigned groups are also the correct groups for these, so leaving them in the PR.

Also cleans up the implementation of the "api_logs" param in the local ci job, which was previously accepting free text

Validation steps

note that an experimental / test only job definition can be found in the first commit on this PR
open up previous runs of the experimental job at https://github.com/HHS/simpler-grants-gov/actions/workflows/e2e-experiment.yml
VERIFY: the first run of the job was run with no tags assigned, and proceeded to run all tests
VERIFY: the first run of the job was run with a single tag assigned, and proceeded to run one test
VERIFY: the first run of the job was run with a two tags assigned, and proceeded to run two tests

If you'd like to test this change yourself, feel free to add the experimental job back and play around with it

aoysimmons-GH · 2026-03-11T20:18:24Z

.github/actions/e2e/action.yml

        npx playwright install --with-deps

-    - name: Run e2e tests (Shard ${{ inputs.shard }}/${{ inputs.total_shards }})
+    - name: Run all e2e tests (Shard ${{ inputs.shard }}/${{ inputs.total_shards }})


................inputs.current_shard }}/${{ inputs.total_shards }})
Missing -current-?

good catch!

aoysimmons-GH · 2026-03-11T20:20:49Z

.github/actions/e2e/action.yml

      shell: bash

+
+    - name: Run e2e tests for ${{ inputs.playwright_tags }} (Shard ${{ inputs.shard }}/${{ inputs.total_shards }})


name: Run e2e tests for ${{ inputs.playwright_tags }} (Shard ${{ inputs.current_shard }}/${{ inputs.total_shards }})

This line also missing -current-

aoysimmons-GH · 2026-03-11T20:34:26Z

.github/workflows/ci-frontend-e2e.yml

+        description: "pipe separated list of @tags denoting groups of tests to run"
+        required: false
        type: string
  workflow_call:


Maybe add these as well?

inputs: api_logs: description: "print api logs on failure?" default: "false" type: string playwright_tags: description: "pipe separated list of @tags denoting groups of tests to run" required: false type: string

aoysimmons-GH · 2026-03-11T20:37:10Z

.github/workflows/ci-frontend-e2e.yml

+        type: choice
+        options:
+          - "true"
+          - "false"


I will use boolean instead:

default: false type: boolean

then update the below :
api_logs: ${{ inputs.api_logs }} to be api_logs: ${{ inputs.api_logs && 'true' || 'false' }}

aoysimmons-GH · 2026-03-11T20:50:16Z

.github/workflows/e2e-staging.yml

@@ -37,12 +41,14 @@ jobs:
      - name: Set target
        run: if [[ -z "${{ inputs.target }}" ]]; then echo "defaulted_target=staging" >> "$GITHUB_ENV"; else echo 'defaulted_target="${{ inputs.target }}"' >> "$GITHUB_ENV"; fi


I know it is not new changes but this seem weird:
maybe fix the ' and "?
run: if [[ -z "${{ inputs.target }}" ]]; then echo "defaulted_target=staging" >> "$GITHUB_ENV"; else echo "defaulted_target=${{ inputs.target }}" >> "$GITHUB_ENV"; fi

aoysimmons-GH · 2026-03-11T20:57:31Z

frontend/tests/e2e/index.spec.ts


-test("has title", async ({ page }) => {
+test("has title", { tag: "@smoke" }, async ({ page }) => {
  await expect(page).toHaveTitle(/Simpler.Grants.gov/);


Maybe fix this one as well. I type in this comment and it is not showing correctly so I attached picture.

aoysimmons-GH · 2026-03-11T21:03:01Z

frontend/tests/e2e/index.spec.ts

-    /https:\/\/github.com\/HHS\/simpler-grants-gov/,
-  );
-});
+test(


I will do this:

test(
"clicking 'follow on GitHub' link opens a new tab pointed at Github repository",
{ tag: "@full-regression" },
async ({ page, context }) => {
const pagePromise = context.waitForEvent("page");
// Click the Follow on GitHub link
await page.getByRole("link", { name: "Follow on GitHub" }).click();
const newPage = await pagePromise;
await expect(newPage).toHaveURL(
/^https://github.com/HHS/simpler-grants-gov(?:/)?$/,
);
},
);

just to confirm, the change here is to change the regex from /https:\/\/github.com\/HHS\/simpler-grants-gov/ to /^https://github.com/HHS/simpler-grants-gov(?:/)?$/, or is there something else?

We should escape the period, but with escaping, I think ^https:\/\/github\.com\/HHS\/simpler-grants-gov(?:\/)\?$ is what you're proposing, and I'm not sure how that is functionally any different from /https:\/\/github\.com\/HHS\/simpler-grants-gov in this case (where we're assuming that the protocol of the url will come at the beginning)

This is what I type but this git comment is not showing what I type in comment

Suggested change

test(

test(

"clicking 'follow on GitHub' link opens a new tab pointed at Github repository",

{ tag: "@full-regression" },

async ({ page, context }) => {

const pagePromise = context.waitForEvent("page");

// Click the Follow on GitHub link

await page.getByRole("link", { name: "Follow on GitHub" }).click();

const newPage = await pagePromise;

await expect(newPage).toHaveURL(

/^https:\/\/github\.com\/HHS\/simpler-grants-gov(?:\/)?$/,

);

},

);

I just learned that I have to put the 'add a suggestion' to put code in comment. This is what I type.

aoysimmons-GH · 2026-03-12T15:14:39Z

frontend/tests/e2e/index.spec.ts

+    const newPage = await pagePromise;
+    await newPage.waitForLoadState();
+    await expect(newPage).toHaveURL(
+      /^https:\/\/github\.com\/HHS\/simpler-grants-gov/,


This one will allow:
https://github.com/HHS/simpler-grants-gov/issues
https://github.com/HHS/simpler-grants-gov?x=1
Anything that starts with that prefix

This one only allows:

Suggested change

/^https:\/\/github\.com\/HHS\/simpler-grants-gov/,

/^https:\/\/github\.com\/HHS\/simpler-grants-gov(?:\/)?$/,

https://github.com/HHS/simpler-grants-gov
https://github.com/HHS/simpler-grants-gov/

got it - this does make this more specific. At this point it should probably just be a string though. I'll make that change

Bhavna-Ramachandran · 2026-03-12T17:26:42Z

Reviewed the changes in this PR and verified that they work as intended.

aoysimmons-GH

Approved

doug-s-nava force-pushed the dschrashun/8184-test-job-tag-groups branch 4 times, most recently from e854e8b to edb8166 Compare March 10, 2026 19:41

doug-s-nava changed the title ~~basic acceptance of tags in jobs~~ [Issue #8184] allow usage of playwright tags in e2e test jobs Mar 10, 2026

doug-s-nava force-pushed the dschrashun/8184-test-job-tag-groups branch 2 times, most recently from f992a5f to a9ec3b6 Compare March 10, 2026 19:52

doug-s-nava marked this pull request as ready for review March 10, 2026 20:00

doug-s-nava added 2 commits March 11, 2026 10:04

basic acceptance of tags in jobs

a08cc17

remove experimental action

9636b66

doug-s-nava force-pushed the dschrashun/8184-test-job-tag-groups branch from a9ec3b6 to 9636b66 Compare March 11, 2026 14:06

aoysimmons-GH reviewed Mar 11, 2026

View reviewed changes

Bhavna-Ramachandran mentioned this pull request Mar 12, 2026

[Issue #8942] Split staging E2E workflow: sharded no-auth, single-worker auth #8965

Open

updates from review

a097c75

aoysimmons-GH requested changes Mar 12, 2026

View reviewed changes

aoysimmons-GH reviewed Mar 12, 2026

View reviewed changes

more review - use hardcoded url string

9eed259

aoysimmons-GH approved these changes Mar 13, 2026

View reviewed changes

doug-s-nava merged commit 16d47e9 into main Mar 13, 2026
17 of 20 checks passed

doug-s-nava deleted the dschrashun/8184-test-job-tag-groups branch March 13, 2026 15:46

		shell: bash


		- name: Run e2e tests for ${{ inputs.playwright_tags }} (Shard ${{ inputs.shard }}/${{ inputs.total_shards }})

		@@ -37,12 +41,14 @@ jobs:
		- name: Set target
		run: if [[ -z "${{ inputs.target }}" ]]; then echo "defaulted_target=staging" >> "$GITHUB_ENV"; else echo 'defaulted_target="${{ inputs.target }}"' >> "$GITHUB_ENV"; fi

-test(
+test(
+  "clicking 'follow on GitHub' link opens a new tab pointed at Github repository",
+  { tag: "@full-regression" },
+  async ({ page, context }) => {
+    const pagePromise = context.waitForEvent("page");
+    // Click the Follow on GitHub link
+    await page.getByRole("link", { name: "Follow on GitHub" }).click();
+    const newPage = await pagePromise;
+    await expect(newPage).toHaveURL(
+      /^https:\/\/github\.com\/HHS\/simpler-grants-gov(?:\/)?$/,
+    );
+  },
+);

	/^https:\/\/github\.com\/HHS\/simpler-grants-gov/,
	/^https:\/\/github\.com\/HHS\/simpler-grants-gov(?:\/)?$/,

Conversation

doug-s-nava commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes proposed

Context for reviewers

Validation steps

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

doug-s-nava Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aoysimmons-GH Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bhavna-Ramachandran commented Mar 12, 2026

Uh oh!

aoysimmons-GH left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

doug-s-nava commented Mar 10, 2026 •

edited

Loading

doug-s-nava Mar 12, 2026 •

edited

Loading

aoysimmons-GH Mar 12, 2026 •

edited

Loading