[Issue #8942] Split staging E2E workflow: sharded no-auth, single-worker auth by Bhavna-Ramachandran · Pull Request #8965 · HHS/simpler-grants-gov

Bhavna-Ramachandran · 2026-03-10T20:43:28Z

Summary

Work for : Task #8942

Changes proposed

Tests that require Login.gov authentication now contain logic to run only against Chrome when targeting staging.
The staging workflow has been split into two jobs:
- e2e-tests-no-auth-sharded : Runs in parallel shards similar to local E2E runs.
- e2e-tests-auth-single-worker : Runs auth-related tests in a single shard to avoid parallel Login.gov sessions.

Context for reviewers

Due to Staging Login.gov limitations - authentication tests cannot run in parallel. Splitting the workflow ensures non-auth tests still benefit from parallelization while auth tests run safely in a single worker.

During this PR, we noticed that - Even with serially run tests, there were few more challenges with Login.gov OTP codes, this PR mainly addresses the fix for these challenges + extra tuning on other tests.

Login.gov OTPs are valid for 30 seconds and single-use. In longer test runs or in serially run tests:
- By the time a Test in series reaches the MFA step, the generated code becomes invalid before login.gov processes it or
- If a test was faster the same code used in previous test login may be attempted again by another test (as the OTP was still valid), which login.gov rejects even if it is still within the same window.

This PR ensures a fresh OTP is generated and handles retry logic if the code is rejected, making the login flow more stable for the E2E test run.

Credit

Tagging approach inspired by @doug-s-nava's PR #8960 , where tags are passed via the test options object instead of embedding them in the test name.

…te artifact naming/merging

…thCurrentURL utility; use domcontentloaded for page.goto to avoid indefinite waits

Bhavna-Ramachandran · 2026-03-12T13:35:57Z

Credit: Tagging approach inspired by @doug-s-nava's PR #8960 , where tags are passed via the test options object instead of embedding them in the test name.

mdragon · 2026-03-13T14:27:21Z

.github/workflows/e2e-create-report.yml

          run-id: ${{ inputs.run_id }}
-          pattern: blob-report-shard-*
+          # Merges both auth and no-auth test artifacts
+          pattern: blob-report*-shard-*


Wouldn't this need to track the change above that names these off the report_name_prefix?

Acknowledged. I have made the update.

mdragon

👍🏻

doug-s-nava

What if we solved this by opting out of anything but chrome within the affected tests rather than updating the jobs?

doug-s-nava · 2026-03-13T18:00:08Z

.github/actions/e2e/action.yml

        npm run test:e2e -- --grep "${{ inputs.playwright_tags }}"
      shell: bash

+    - name: Run e2e tests excluding ${{ inputs.playwright_tags_invert }} (Shard ${{ inputs.current_shard }}/${{ inputs.total_shards }})


with this in here there's a chance that if a user supplies both a positive list and a negative list, the same tests could run twice.

I think the ability to run a negative list is not a super high priority and outside the scope of this change, so I'd like to save implementing this to a separate PR if that's ok

Yes I will remove this from action and handle the exclusion directly in staging workflow. Will make this update now.

doug-s-nava · 2026-03-13T18:05:36Z

.github/workflows/e2e-staging.yml

          total_shards: ${{ matrix.total_shards }}
          current_shard: ${{ matrix.shard }}
-          playwright_tags: ${{ inputs.playwright_tags }}
+          playwright_tags_invert: "@auth"


this isn't quite what I had in mind for this tag. The @auth tag in my conception is for tests of login specific behavior, or behavior that is tied to specific permissions, not just any functionality that requires the user to be logged in. It's not a big deal since we haven't really started implementing tags yet, but if we're using @auth to tag any test where a user logs in we may need to define a different tag to track tests of auth specific behavior

Based on our discussion. I am removing the auth tag.

doug-s-nava · 2026-03-13T18:12:15Z

frontend/tests/e2e/utils/authenticate-e2e-user-utils.ts

    throw new Error(`Unsupported env ${targetEnv}`);
  }

  if (isMobile) {


I know this is out of scope but why do we want to always open the mobile nav after logging in?

In some login tests, post login the flow is expected to open mobile nav (on mobile browser it is hidden behind hamburger menu) if needed, since this function is a shared login helper, we have it part of this. Based on need we can also move it part of each test too.

doug-s-nava · 2026-03-13T18:13:25Z

frontend/tests/e2e/utils/create-application-utils.ts

+/**
+ * Navigates to a URL with retry logic to handle transient network errors.
+ */
+async function gotoWithRetry(


this seems like a general util rather than something tied to "create-application". Can we move to a more general file?

Acknowledged. I have moved this to lifecycle-utils

doug-s-nava · 2026-03-13T18:14:47Z

frontend/tests/e2e/utils/perform-login-utils.ts

 const TIMEOUT_HOME = playwrightEnv.targetEnv === "staging" ? 180000 : 60000;
 const TIMEOUT_MFA = 120000;

+// TOTP codes are valid for 30s windows. If we're within this many seconds


doug-s-nava · 2026-03-13T18:15:38Z

frontend/tests/e2e/utils/perform-login-utils.ts

+    'button:has-text("Sign out"), a:has-text("Sign out")',
+  );
+  if (await existingSignOut.isVisible({ timeout: 3000 }).catch(() => false)) {
+    // console.log("performStagingLogin: already logged in, skipping login flow");


can remove this

acknowledged.

doug-s-nava · 2026-03-13T18:16:53Z

frontend/tests/e2e/saved-opportunities.spec.ts

-test.afterEach(async ({ context }) => {
-  await context.close();
-});
+// Note: do NOT close context in afterEach — Playwright manages context lifecycle


don't think we need this comment here - if this should be documented let's put it somewhere more general. Should we start a document for things like this either in /documentation or in Confluence?

Yes it is useful to document this. I saw failures in this test in Staging and I made this note. If its ok I would suggest to keep this in this spec too, it would serve as reminder when we make updates or debug this flow. Let me know your thoughts.

doug-s-nava · 2026-03-13T18:20:36Z

.github/workflows/e2e-staging.yml

+        with:
+          version: ${{ inputs.version || github.ref }}
+          target: ${{ env.defaulted_target }}
+          total_shards: 1


I think this is still going to run all browsers, just in 1 shard. The configurations are defined in the playwright config so unless we're overriding that or supplying a new config this doesn't really solve the problem

Yes, you are right. However, the browser execution is controlled in the test command we pass, where we had explicitly specified the Chrome project:
npx playwright test --config ./tests/playwright.config.ts --project=Chrome --workers=1 --reporter=list --grep @auth
So the auth workflow ran only against the Chrome project with a single worker, which avoided the parallel session in staging.

Bhavna-Ramachandran changed the title ~~Split staging E2E workflow: sharded no-auth, single-worker auth; upda…~~ [Issue #8942] Split staging E2E workflow: sharded no-auth, single-worker auth Mar 11, 2026

Bhavna-Ramachandran added 5 commits March 11, 2026 19:28

Split staging E2E workflow: sharded no-auth, single-worker auth; upda…

ea7334d

…te artifact naming/merging

E2E: tag-based auth split, robust form nav, artifact/report improvements

30e8f2a

Fix lint errors and update overall auth E2E workflow

e77dd1a

E2E: tag-based auth split, robust form nav, artifact/report improvements

d6c1217

Fix lint errors and update overall auth E2E workflow

60bb632

Bhavna-Ramachandran force-pushed the BR-8942-E2E-Staging-Run branch from c851be6 to 60bb632 Compare March 11, 2026 19:28

Bhavna-Ramachandran added 4 commits March 11, 2026 20:18

Updated with already logged in guard and resolved lint errors

69bd321

Add @auth tag to the new SFLLL form test

b64b674

Resolve flakiness: remove redundant networkidle wait in refreshPageWi…

bc63d76

…thCurrentURL utility; use domcontentloaded for page.goto to avoid indefinite waits

Add waiting for stable app state

7483cc8

Bhavna-Ramachandran added 2 commits March 12, 2026 14:05

Fix: stabilize login redirect and org dropdown loading on staging

2a18616

Resolve lint errors

42857c2

Bhavna-Ramachandran requested a review from doug-s-nava March 12, 2026 15:42

Bhavna-Ramachandran mentioned this pull request Mar 12, 2026

Run staging targeted e2e tests with login only on chrome #8942

Open

1 task

mdragon reviewed Mar 13, 2026

View reviewed changes

Bhavna-Ramachandran added 2 commits March 13, 2026 17:09

Restore workflow files to main branch versions

dce8a22

Merge branch 'main' into BR-8942-E2E-Staging-Run

35faa3c

mdragon previously approved these changes Mar 13, 2026

View reviewed changes

doug-s-nava reviewed Mar 13, 2026

View reviewed changes

Address review comments and made updates post discussion

ee4972c

Bhavna-Ramachandran dismissed mdragon’s stale review via ee4972c March 13, 2026 20:07

Address review comments and clean up annotations

38be1d0

Conversation

Bhavna-Ramachandran commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes proposed

Context for reviewers

Credit

Uh oh!

Bhavna-Ramachandran commented Mar 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdragon left a comment

Choose a reason for hiding this comment

Uh oh!

doug-s-nava left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Bhavna-Ramachandran commented Mar 10, 2026 •

edited

Loading