chore(compass-e2e-tests): add assistant end to end tests COMPASS-9748 #7429

gagik · 2025-10-07T19:02:14Z

Adds e2e tests to the core assistant features.

Copilot

Pull Request Overview

This PR adds comprehensive end-to-end tests for the MongoDB Assistant feature in Compass. The tests cover various scenarios including AI feature opt-in flow, message sending, drawer interactions, and entry points from explain plans and error messages.

Adds complete e2e test suite for Assistant functionality with 593 lines of test code
Implements mock assistant service to simulate AI responses with streaming support
Adds necessary test selectors and data-testid attributes for UI elements

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
packages/compass-generative-ai/src/components/ai-optin-modal.tsx	Adds data-testid for testing the AI opt-in modal
packages/compass-e2e-tests/tests/assistant.test.ts	Complete test suite covering assistant drawer, opt-in flow, messaging, and entry points
packages/compass-e2e-tests/helpers/selectors.ts	Adds selectors for assistant UI elements and AI settings
packages/compass-e2e-tests/helpers/assistant-service.ts	Mock service implementation for simulating assistant API responses

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

packages/compass-e2e-tests/tests/assistant.test.ts

packages/compass-e2e-tests/helpers/assistant-service.ts

codeowners-service-app · 2025-10-08T10:45:42Z

Assigned mabaasit for team compass-developers because johnjackweir is out of office.

gagik · 2025-10-08T10:44:37Z

packages/compass-e2e-tests/tests/assistant.test.ts

+
+  describe('drawer visibility', function () {
+    it('shows the assistant drawer button when AI features are enabled', async function () {
+      await setAIFeatures(browser, true);


so hypothetically this function should work and it does in one direction but after opting in, going back to opt-out isn't possible. Basically as a result the tests are order-dependent which isn't great. I think it's because of some of our logic and possibly related to the more general issue with i.e. feature flags we were seeing.

@lerouxb suggested

compass/packages/compass-e2e-tests/tests/time-to-first-query.test.ts

Line 56 in b9a6fde

compass = await init(this.test?.fullTitle(), { firstRun: false });

. Will look into this.

If that doesn't work then I guess we can also add a hidden way to opt-out again.

Either we can just tweak settings so it can be overridden or we can add some custom event like we did for selecting a file:

compass/packages/compass-import-export/src/components/export-modal.tsx

Lines 224 to 240 in b9a6fde

useEffect(() => {

if (isOpen) {

// For e2e testing we can't set the value of a file output

// for security reasons, so we listen to a dom event that sets it.

// https://github.com/electron-userland/spectron/issues/23

document.addEventListener(

'selectExportFileName',

onSelectExportFileNameEvent

);

return () => {

document.removeEventListener(

'selectExportFileName',

onSelectExportFileNameEvent

);

};

}

}, [isOpen, onSelectExportFileNameEvent]);

or what we did with test:show-connect-window

compass/packages/compass/src/main/window-manager.ts

Line 272 in b9a6fde

'test:show-connect-window': () => void showConnectWindow(compassApp),

or what we did to get things like the logs:

compass/packages/compass-e2e-tests/helpers/compass.ts

Lines 226 to 229 in b9a6fde

ipcRenderer.invoke('compass:logPath'),

ipcRenderer.invoke('compass:userDataPath'),

ipcRenderer.invoke('compass:appName'),

ipcRenderer.invoke('compass:mainProcessPid'),

I don't know which one is the right way I'm just saying there are probably options if we need them.

or see browser.getFeature() and browser.setFeature() For example

compass/packages/compass-web/src/preferences.tsx

Lines 14 to 51 in b9a6fde

const kSandboxUpdateFn = Symbol.for('@compass-web-sandbox-update-preferences');

/**

* Only used in the sandbox to provide a way to update preferences.

* @internal

*/

export const SandboxPreferencesUpdateProvider = ({

children,

}: {

children: React.ReactNode;

}) => {

const [updateTrigger] = useState<SandboxPreferencesUpdateTrigger>(() => {

return (

updatePreferencesFn: (

preferences: Partial<AllPreferences>

) => Promise<AllPreferences>

) => {

// eslint-disable-next-line no-console

console.info(

`[compass-web sandbox] call window[Symbol.for('@compass-web-sandbox-update-preferences')]({}) to dynamically update preferences`

);

(globalThis as any)[kSandboxUpdateFn] = (

preferences: Partial<AllPreferences>

) => {

return updatePreferencesFn(preferences);

};

return () => {

delete (globalThis as any)[kSandboxUpdateFn];

};

};

});

return (

<SandboxPreferencesUpdateTriggerContext.Provider value={updateTrigger}>

{children}

</SandboxPreferencesUpdateTriggerContext.Provider>

);

};

browser.setFeature is what this one uses behind the scenes, doesn't seem to fully do the trick so will just do the hard reset

gagik · 2025-10-08T10:49:40Z

packages/compass-e2e-tests/tests/assistant.test.ts

+  }
+}
+
+async function getDisplayedMessages(browser: CompassBrowser) {


I originally defined these functions in hooks but it was making the code too dirty so I moved them out here and made it pass browser instead... I imagine if chat becomes popular enough we can extend browser itself.

That's fine - I usually add things locally and then only factor them out once necessary. I think that's a good way to work.

lerouxb · 2025-10-08T11:22:52Z

packages/compass-e2e-tests/tests/assistant.test.ts

+      ]);
+    });
+
+    it('can copy assistant message to clipboard', async function () {


I'm assuming that for now this one is not going to work for the web tests. (we also run the sandbox in the browser if I remember correctly and there might be ones that hit DE - I'm a bit out of the loop).

lerouxb · 2025-10-08T11:23:10Z

packages/compass-e2e-tests/tests/assistant.test.ts

+
+          mockAssistantServer.setResponse({
+            status: 200,
+            body: 'You should create an index.',


lerouxb · 2025-10-08T11:28:26Z

packages/compass-e2e-tests/tests/assistant.test.ts

+  await browser.setFeature('optInGenAIFeatures', enabled);
+
+  // Wait for the IPC to be processed
+  await browser.pause(500);


This kind of pause is kinda dodgy and shouldn't be necessary because anything depending on the result should/would be waiting for whatever it is expecting to happen anyway.

lerouxb · 2025-10-08T11:29:26Z

packages/compass-e2e-tests/tests/assistant.test.ts

+  const drawerButton = browser.$(Selectors.AssistantDrawerButton);
+  await drawerButton.waitForDisplayed();
+  await drawerButton.waitForClickable();
+  await drawerButton.click();


Just FYI in theory await browser.clickVisible(Selectors.AssistantDrawerButton); does all this and more.

Would have been better if we added it as a custom element command rather than a browser command, but it is there.

lerouxb · 2025-10-08T11:32:05Z

packages/compass-e2e-tests/tests/assistant.test.ts

+
+async function getDisplayedMessages(browser: CompassBrowser) {
+  // Wait for the messages container to be visible
+  const chatMessages = browser.$(Selectors.AssistantChatMessages);


Super minor style nit (and I don't think we're very consistent about it), but if you're not going to use the chatMessages var again you can also just chain like await browser.$(Selectors.AssistantChatMessages).waitForDisplayed();

lerouxb · 2025-10-08T11:35:17Z

packages/compass-e2e-tests/tests/assistant.test.ts

+  const explainModal = browser.$(Selectors.AggregationExplainModal);
+  await explainModal.waitForDisplayed();
+
+  // Wait for the explain plan to be ready (loader should disappear)


I would personally remove this whole block because the loader can appear/disappear instantly or it can happen within one of the intervals that any wait command polls. ie. it is going to be flaky. Plus it also suffers from the problem that the loader starts off not being there.

You can just wait for the interpret button to be there and click it - it should be the same result anyway.

lerouxb · 2025-10-08T11:36:25Z

packages/compass-e2e-tests/tests/assistant.test.ts

+  await explainModal.waitForDisplayed({ reverse: true });
+
+  // The assistant drawer should open
+  const assistantDrawer = browser.$(Selectors.SideDrawer);


Can we not rather wait for the assistant drawer section specifically? In case over time we also end up with the Data Modelling section in there.

lerouxb · 2025-10-08T11:37:44Z

packages/compass-e2e-tests/tests/assistant.test.ts

+}
+
+async function useErrorViewEntryPoint(browser: CompassBrowser) {
+  const connectionToastErrorDebugButton = browser.$(


I know I mentioned this above too but I really think you can just await browser.clickVisible(Selectors.ConnectionToastErrorDebugButton); which does most of this. I suppose you can wait for the toast to disappear afterwards if you want.

(This is nitpicking, btw. What you have here is fine.)

lerouxb · 2025-10-08T11:39:54Z

packages/compass-e2e-tests/helpers/assistant-service.ts

+  sendChunk();
+}
+
+export async function startMockAssistantServer(


So.. Usually E2E tests are purposefully integration tests, ie. we try not to mock or stub anything if at all possible. But I do get that the assistant is an expensive feature to be running many times on every PR and it is also particularly non-deterministic.

So I think in this case it is OK, just something to keep in mind going forward.

gagik added 3 commits October 7, 2025 19:19

chore(compass-e2e-tests): add assistant end to end tests COMPASS-9384

a656e36

chore: add entry points

4d1cddc

chore: fixup

bd6fc2f

gagik force-pushed the gagik/e2e branch from ad314e3 to bd6fc2f Compare October 7, 2025 19:04

chore: cleanup and fix

dccb354

gagik changed the title ~~chore(compass-e2e-tests): add assistant end to end tests COMPASS-9384~~ chore(compass-e2e-tests): add assistant end to end tests COMPASS-9748 Oct 8, 2025

gagik marked this pull request as ready for review October 8, 2025 10:41

gagik requested a review from a team as a code owner October 8, 2025 10:41

gagik requested review from johnjackweir and Copilot October 8, 2025 10:41

Copilot AI reviewed Oct 8, 2025

View reviewed changes

packages/compass-e2e-tests/tests/assistant.test.ts Outdated Show resolved Hide resolved

packages/compass-e2e-tests/helpers/assistant-service.ts Show resolved Hide resolved

codeowners-service-app bot requested a review from mabaasit October 8, 2025 10:45

chore: remove redundant comments

1010161

gagik commented Oct 8, 2025

View reviewed changes

gagik requested a review from lerouxb October 8, 2025 10:47

gagik commented Oct 8, 2025

View reviewed changes

lerouxb reviewed Oct 8, 2025

View reviewed changes

	useEffect(() => {
	if (isOpen) {
	// For e2e testing we can't set the value of a file output
	// for security reasons, so we listen to a dom event that sets it.
	// https://github.com/electron-userland/spectron/issues/23
	document.addEventListener(
	'selectExportFileName',
	onSelectExportFileNameEvent
	);
	return () => {
	document.removeEventListener(
	'selectExportFileName',
	onSelectExportFileNameEvent
	);
	};
	}
	}, [isOpen, onSelectExportFileNameEvent]);

	ipcRenderer.invoke('compass:logPath'),
	ipcRenderer.invoke('compass:userDataPath'),
	ipcRenderer.invoke('compass:appName'),
	ipcRenderer.invoke('compass:mainProcessPid'),

	const kSandboxUpdateFn = Symbol.for('@compass-web-sandbox-update-preferences');

	/**
	* Only used in the sandbox to provide a way to update preferences.
	* @internal
	*/
	export const SandboxPreferencesUpdateProvider = ({
	children,
	}: {
	children: React.ReactNode;
	}) => {
	const [updateTrigger] = useState<SandboxPreferencesUpdateTrigger>(() => {
	return (
	updatePreferencesFn: (
	preferences: Partial<AllPreferences>
	) => Promise<AllPreferences>
	) => {
	// eslint-disable-next-line no-console
	console.info(
	`[compass-web sandbox] call window[Symbol.for('@compass-web-sandbox-update-preferences')]({}) to dynamically update preferences`
	);
	(globalThis as any)[kSandboxUpdateFn] = (
	preferences: Partial<AllPreferences>
	) => {
	return updatePreferencesFn(preferences);
	};
	return () => {
	delete (globalThis as any)[kSandboxUpdateFn];
	};
	};
	});

	return (
	<SandboxPreferencesUpdateTriggerContext.Provider value={updateTrigger}>
	{children}
	</SandboxPreferencesUpdateTriggerContext.Provider>
	);
	};

chore(compass-e2e-tests): add assistant end to end tests COMPASS-9748 #7429

Are you sure you want to change the base?

chore(compass-e2e-tests): add assistant end to end tests COMPASS-9748 #7429

Conversation

gagik commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

codeowners-service-app bot commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lerouxb Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gagik commented Oct 7, 2025 •

edited

Loading

lerouxb Oct 8, 2025 •

edited

Loading