Add Cypress testing framework and update Kubernetes configuration

tests/browser/README.md

@@ -1,3 +1,183 @@
+The `cypress/` directory holds Cypress tests and the `tests/` directory holds Playwright tests.
+
+The following upstream projects have Playwright tests
+
+* JupyterLab (https://github.com/jupyterlab/jupyterlab/tree/main/galata)
+* code-server (https://github.com/coder/code-server/tree/main/test)
+
+Honorable mentions include
+
+* VSCode uses custom framework where Playwright is one of the possible runners (https://github.com/microsoft/vscode/wiki/Writing-Tests)
+* RStudio components have Playwright tests (https://github.com/rstudio/shinyuieditor, https://github.com/rstudio/xterm.js)
+* Some RStudio tests are implemented in private repository https://github.com/rstudio/rstudio/issues/10400, possibly in R https://github.com/rstudio/rstudio/tree/main/src/cpp/tests/automation with https://github.com/rstudio/chromote)
+
+The following upstream projects have Cypress tests
+
+* Elyra (https://github.com/elyra-ai/elyra/tree/main/cypress)
+* ODH Dashboard (https://github.com/opendatahub-io/odh-dashboard/tree/main/frontend/src/__tests__/cypress)
+
+# Cypress
+
+The Cypress part was added after the Playwright part below.
+Therefore, we are starting with an existing pnpm project folder.
+
+```shell
+pnpm add --save-dev cypress
+```
+
+Since pnpm skips running build scripts by default, just run `cypress install` manually.
+
+```
+╭ Warning ───────────────────────────────────────────────────────────────────────────────────╮
+│                                                                                            │
+│   Ignored build scripts: cypress.                                                          │
+│   Run "pnpm approve-builds" to pick which dependencies should be allowed to run scripts.   │
+│                                                                                            │
+╰────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
+```shell
+pnpm cypress install
+```
+
+## Getting started
+
+> https://learn.cypress.io/testing-your-first-application/installing-cypress-and-writing-your-first-test
+
+Cypress operates in two modes,
+the noninteractive `run` mode and the interactive `open` mode that is useful for development.
+
+```shell
+pnpm cypress run
+pnpm cypress open
+```
+
+To specify base URL, set the environment variable.
+
+```shell
+BASE_URL=https://nb_name.apps.oc_domain/notebook/ns_name/nb_name pnpm cypress open --e2e --browser chrome
+```
+
+Upon first run, `cypress open` will ask to begin with either E2E or Component testing.
+Choose E2e, and the following files are created if they did not exist before:
+
+* `cypress.config.ts`: The Cypress config file for E2E testing.
+* `cypress/support/e2e.ts`: The support file that is bundled and loaded before each E2E spec.
+* `cypress/support/commands.ts`: A support file that is useful for creating custom Cypress commands and overwriting existing ones.
+* `cypress/fixtures/example.json`: Added an example fixtures file/folder.
+
+For any subsequent run, Cypress offers a choice of three test environments:
+
+1. Chrome
+2. Electron
+3. Firefox
+
+Pick Chrome and click `Start E2E Testing in Chrome` to confirm.
+
+If there are no tests (specs) detected, Cypress offers to `Scaffold example specs` or to `Create new spec`.
+To experience this and maybe experiment with example specs,
+temporarily delete everything under `cypress/e2e/` and let Cypress refresh.
+
+## Developing tests
+
+Start `cypress open` in E2E mode with Chrome
+
+```shell
+BASE_URL=... pnpm cypress open --e2e --browser chrome
+```
+
+The `open` mode can be further enhanced by enabling the (currently experimental) Cypress Studio.
+
+Use this to quickly scaffold the test steps and then refactor them to use page objects.
+
+* https://docs.cypress.io/app/guides/cypress-studio
+* https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/
+* https://docs.cypress.io/app/core-concepts/best-practices#Organizing-Tests-Logging-In-Controlling-State
+
+```typescript
+// cypress.config.ts
+import { defineConfig } from 'cypress'
+
+export default defineConfig({
+  e2e: {
+    experimentalStudio: true,
+  },
+})
+```
+
+## Execution model
+
+Cypress execution model can be tricky.
+
+Do read the introductory docs page, then the retry-ability,
+and then the conditional testing page to appreciate the ramifications.
+
+* https://docs.cypress.io/app/core-concepts/introduction-to-cypress
+* https://docs.cypress.io/app/core-concepts/retry-ability
+* https://docs.cypress.io/app/guides/conditional-testing
+
+Cypress is not a general purpose web browser automation framework,
+that was sufficiently clarified in the introduction docs, and also read the following.
+
+* https://docs.cypress.io/app/references/trade-offs
+* https://docs.cypress.io/app/guides/cross-origin-testing
+
+Also do check out:
+
+* https://docs.cypress.io/app/core-concepts/best-practices
+
+## Problems and how to solve them
+
+See above for the execution model notes, and the Cypress trade-offs documentation.
+
+### Browser runs out of memory
+
+Often, the `cypress open` browser crashes with the following error message.
+
+```
+We detected that the Chrome Renderer process just crashed.
+
+We have failed the current spec but will continue running the next spec.
+
+This can happen for a number of different reasons.
+
+If you're running lots of tests on a memory intense application.
+  - Try increasing the CPU/memory on the machine you're running on.
+  - Try enabling experimentalMemoryManagement in your config file.
+  - Try lowering numTestsKeptInMemory in your config file during 'cypress open'.
+
+You can learn more here:
+
+https://on.cypress.io/renderer-process-crashed
+```
+
+The advice helps somewhat, but Elyra still keeps crashing from time to time in `cypress open`.
+
+### Cross-origin testing
+
+Prior to Cypress 14, the [`document.domanin`](https://developer.mozilla.org/en-US/docs/Web/API/Document/domain) would be automatically set by Cypress.
+Now that it is no loger true, it is as the documentation says:
+
+> You can visit two or more origins in different tests without needing cy.origin().
+> (https://docs.cypress.io/app/guides/cross-origin-testing#What-Cypress-does-under-the-hood)
+
+This is especially annoying when Dashboard, Workbench,
+and OAuth server each live in a separate origin and one test needs to visit all three.
+
+#### Solutions for cross-origin testing
+
+* The origin for each test is pinned by wherever the first `cy.visit()` ends up going, taking redirects into account.
+  * Always `cy.visit()` first the origin where the test needs to spend the most time.
+* Use `cy.origin()` when needed. Beware that custom commands don't work on secondary origins unless `Cypress.require()` (experimental) is called!
+* Reconfigure oauth-proxy to allow bearer token authentication, or skip auth altogether and expose workbench container directly.
+  * https://github.com/openshift/oauth-proxy/issues/179#issuecomment-1202279241
+  * https://github.com/openshift/oauth-proxy/blob/8d8daec87683f43a15c1d74f05cb0f2635dba04e/main.go#L76
+* Write the tests so that only one origin needs to be touched in the test.
+  * `cy.session()` can hold login cookies established in a `before` step.
+  * `cy.request()` is not bound by origin restrictions, attempt to log in through API.
+
+# Playwright
+
 This is a basic Playwright in Typescript that was setup like this
 
 ```shell
@@ -51,3 +231,6 @@ CI captures execution traces that can be opened in [the trace viewer](https://pl
 pnpm playwright show-trace path/to/trace.zip
 ```
 
+## Good practices
+
+* https://playwright.dev/docs/best-practices

tests/browser/cypress.config.ts

@@ -0,0 +1,31 @@
+// https://docs.cypress.io/app/references/configuration
+
+import { defineConfig } from "cypress";
+import * as testConfig from "./cypress/utils/testConfig.js";
+
+export default defineConfig({
+  e2e: {
+    env: {
+        LOGIN: !!process.env.LOGIN,
+    },
+    baseUrl: testConfig.BASE_URL,
+    // this helps with browser crashes
+    experimentalMemoryManagement: true,
+    numTestsKeptInMemory: 1,
+
+    // cross-origin testing
+
+    // disable webSecurity
+    chromeWebSecurity: false,
+    // Using require() or import() within the cy.origin() callback is not supported.
+    // Use Cypress.require() to include dependencies instead,
+    // but note that it currently requires enabling the experimentalOriginDependencies flag.
+    experimentalOriginDependencies: true,
+    // https://docs.cypress.io/app/guides/cypress-studio
+    experimentalStudio: false,
+    // https://docs.cypress.io/app/plugins/plugins-guide
+    setupNodeEvents(on, config) {
+      // implement node event listeners here
+    },
+  },
+});

tests/browser/cypress/fixtures/example.json

@@ -0,0 +1,5 @@
+{
+  "name": "Using fixtures to represent data",
+  "email": "[email protected]",
+  "body": "Fixtures are a great way to mock data for responses to routes"
+}

tests/browser/cypress/support/commands.ts

@@ -0,0 +1,131 @@
+/// <reference types="cypress" />
+/// <reference types="@testing-library/cypress" />
+// ***********************************************
+// This example commands.ts shows you how to
+// create various custom commands and overwrite
+// existing commands.
+//
+// For more comprehensive examples of custom
+// commands please read more here:
+// https://on.cypress.io/custom-commands
+// ***********************************************
+//
+//
+// -- This is a parent command --
+// Cypress.Commands.add('login', (email, password) => { ... })
+//
+//
+// -- This is a child command --
+// Cypress.Commands.add('drag', { prevSubject: 'element'}, (subject, options) => { ... })
+//
+//
+// -- This is a dual command --
+// Cypress.Commands.add('dismiss', { prevSubject: 'optional'}, (subject, options) => { ... })
+//
+//
+// -- This will overwrite an existing command --
+// Cypress.Commands.overwrite('visit', (originalFn, url, options) => { ... })
+//
+// declare global {
+//   namespace Cypress {
+//     interface Chainable {
+//       login(email: string, password: string): Chainable<void>
+//       drag(subject: string, options?: Partial<TypeOptions>): Chainable<Element>
+//       dismiss(subject: string, options?: Partial<TypeOptions>): Chainable<Element>
+//       visit(originalFn: CommandOriginalFn, url: string, options: Partial<VisitOptions>): Chainable<Element>
+//     }
+//   }
+// }
+
+// https://youtrack.jetbrains.com/issue/AQUA-3175/Command-click-expects-element-as-a-previous-subject-but-any-is-passed-from-wrap-command
+
+// https://docs.cypress.io/api/cypress-api/custom-commands
+
+Cypress.Commands.add('establishOrigin', (url: string) => {
+    // https://stackoverflow.com/questions/72197730/removing-an-intercept-in-cypress
+    // https://github.com/cypress-io/cypress/issues/23192#issuecomment-2401901751
+    cy.intercept(
+            {
+                method: 'GET',
+                url: url,
+                times: 1
+            },
+            {
+                body: '<html lang="en"></html>',
+                headers: {
+                    'content-type': 'text/html'
+                },
+            })
+        .as('establishOrigin');
+    cy.visit(url);
+    cy.wait('@establishOrigin');
+})
+
+Cypress.Commands.add('fill',
+    { prevSubject: 'element' }, (subject, text, options) => {
+    cy.wrap(subject).clear();
+    return cy.wrap(subject).type(text, options);
+});
+
+Cypress.Commands.add('visitWithLogin', (relativeUrl, credentials = {USERNAME: 'admin-user', PASSWORD: 'xxx'}) => {
+    if (Cypress.env('MOCK')) {
+        cy.visit(relativeUrl);
+    } else {
+        let fullUrl: string;
+        // if (relativeUrl.replace(/\//g, '')) {
+        //     fullUrl = new URL(relativeUrl, Cypress.config('baseUrl') || '').href;
+        // } else {
+        //     fullUrl = new URL(Cypress.config('baseUrl') || '').href;
+        // }
+        fullUrl = relativeUrl;
+        cy.step(`Navigate to: ${fullUrl}`);
+        cy.intercept('GET', fullUrl, { log: false }).as('page');
+        cy.visit(fullUrl, { failOnStatusCode: false });
+        cy.wait('@page', { log: false }).then((interception) => {
+            const statusCode = interception.response?.statusCode;
+            if (statusCode === 403 || statusCode === 302) {
+                cy.log('Log in');
+                // cy.get('form[action="/oauth/start"]').submit();
+
+                cy.intercept('GET', fullUrl, { log: false }).as('otherpage');
+                cy.get('form').submit();
+                cy.wait('@otherpage', { log: false }).then((interception) => {
+                    const statusCode = interception.response?.statusCode;
+                    if (statusCode === 403 || statusCode === 302) {
+                        cy.visit(fullUrl, {failOnStatusCode: false});
+                    } else if (!interception.response || statusCode !== 200) {
+                        console.log("aaaa")
+                        console.log(interception.response);
+                        throw new Error(`Failed to visit '${fullUrl}'. Status code: ${statusCode || 'unknown'}`);
+                    }
+                });
+            } else if (!interception.response || statusCode !== 200) {
+                console.log("aaaa")
+                console.log(interception.response);
+                throw new Error(`Failed to visit '${fullUrl}'. Status code: ${statusCode || 'unknown'}`);
+            }
+        });
+    }
+});
+
+declare global {
+    namespace Cypress {
+        interface Chainable<Subject = any> {
+            /**
+             * `clear()`s the current contents and then `type()`s in the given `text`
+             */
+            fill(text: string, options?: Partial<TypeOptions>): Cypress.Chainable<JQueryWithSelector>
+
+            /**
+             * Sets the default origin for the test to `url`.
+             *
+             * Workbenches reply to unauthed requests with a 302 redirect to oauth page on a different origin.
+             * Therefore, a test will end up with the origin of oauth as its main origin.
+             * Every interaction with the workbench after logging in would have to be wrapped in a `cy.origin()` block.
+             *
+             * This can be overcome if we mock the initial visit to return 200 and establish the origin.
+             */
+            establishOrigin(url: string): void
+        }
+    }
+}

tests/browser/cypress/support/e2e.ts

@@ -0,0 +1,22 @@
+// ***********************************************************
+// This example support/e2e.ts is processed and
+// loaded automatically before your test files.
+//
+// This is a great place to put global configuration and
+// behavior that modifies Cypress.
+//
+// You can change the location of this file or turn off
+// automatically serving support files with the
+// 'supportFile' configuration option.
+//
+// You can read more here:
+// https://on.cypress.io/configuration
+// ***********************************************************
+
+// https://github.com/filiphric/cypress-plugin-steps
+import 'cypress-plugin-steps'
+// https://testing-library.com/docs/cypress-testing-library/intro/
+import '@testing-library/cypress/add-commands'
+
+// Import commands.js using ES2015 syntax:
+import './commands'

tests/browser/cypress/utils/testConfig.ts

@@ -0,0 +1,14 @@
+import process from "node:process";
+import fs from "fs";
+
+import YAML from 'yaml';
+
+class TestConfig {
+    ODH_DASHBOARD_URL?: string;
+}
+
+export const testConfig: TestConfig | undefined = process.env.CY_TEST_CONFIG
+    ? YAML.parse(fs.readFileSync(process.env.CY_TEST_CONFIG).toString())
+    : undefined;
+
+export const BASE_URL = testConfig?.ODH_DASHBOARD_URL || process.env.BASE_URL || '';