Skip to content

fix(amazonq): Addition of wait function for Amazon Q inline chat UI E2E Tests and Inline Keybind Shortcut Test #7840

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: feature/ui-e2e-tests
Choose a base branch
from
Open
28 changes: 17 additions & 11 deletions packages/amazonq/test/e2e_new/amazonq/tests/inline.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import '../utils/setup'
import { Workbench, EditorView, InputBox, TextEditor, WebviewView, Key } from 'vscode-extension-tester'
import { testContext } from '../utils/testContext'
import { pressKey, createNewTextFile, writeToTextEditor } from '../utils/generalUtils'
import { createNewTextFile, writeToTextEditor, waitForEditorStabilization } from '../utils/generalUtils'
import assert from 'assert'

describe('Amazon Q Inline Completion / Chat Functionality', function () {
Expand All @@ -20,25 +20,31 @@ describe('Amazon Q Inline Completion / Chat Functionality', function () {
webviewView = testContext.webviewView
await webviewView.switchBack()
workbench = testContext.workbench

editorView = new EditorView()
testContext.editorView = editorView

textEditor = await createNewTextFile(workbench, editorView)
})

it('Inline Test', async () => {
await writeToTextEditor(textEditor, 'Select Me')
after(async function () {
// Switch back to Webview Iframe when dealing with external webviews from Amazon Q.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should be corrected to iframe

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha!

await editorView.closeAllEditors()
await webviewView.switchToFrame()
})
it('Inline Test Shortcut', async () => {
await writeToTextEditor(textEditor, 'def factorial(n):')
const text = await textEditor.getText()
assert.equal(text, 'Select Me')
assert.equal(text, 'def factorial(n): ')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe a silly question, but why does this pass with the space at the end, if the writeToTextEditor above doesn't add a space at the end? Or does it automatically?

Is this related to the dummy space below?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! The dummy space actually creates a space at the end of the word, and then it types from before that space so the space ends up at the end of whatever text we generate. I actually removed this logic since it is no longer needed in the test (the assert equal text)

await textEditor.clearText()

const textBefore = await textEditor.getText()
await workbench.executeCommand('Amazon Q: Inline Chat')
const input = new InputBox()
await input.sendKeys('Write a simple sentece')
await input.sendKeys('Generate the fibonacci sequence through iteration')
await input.sendKeys(Key.ENTER)
const driver = textEditor.getDriver()
await pressKey(driver, 'ENTER')
await pressKey(driver, 'TAB')
// Wait for Amazon Q to finish generating code
await waitForEditorStabilization(textEditor)

const textAfter = await textEditor.getText()
assert(textAfter.length > textBefore.length, 'Amazon Q generated code')
await textEditor.clearText()
})
})
37 changes: 35 additions & 2 deletions packages/amazonq/test/e2e_new/amazonq/utils/generalUtils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,42 @@ export async function createNewTextFile(workbench: Workbench, editorView: Editor
* @returns Promise<void>
*/
export async function writeToTextEditor(textEditor: TextEditor, text: string): Promise<void> {
// We require a "dummy" space to be written such that we can properly index the
// number of lines to register the textEditor.
await textEditor.typeTextAt(1, 1, ' ')
const currentLines = await textEditor.getNumberOfLines()
const nextLine = currentLines + 1
await textEditor.typeTextAt(nextLine, 1, text)
await textEditor.typeTextAt(currentLines, 1, text)
}

/**
* Waits for editor content to stabilize by checking if line count stops changing
* @param editor The TextEditor instance
* @param timeout Maximum time to wait in milliseconds (default: 15000)
* @returns Promise<void>
* @throws Error if timeout is exceeded
*/
export async function waitForEditorStabilization(editor: TextEditor, timeout = 15000): Promise<void> {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a better name for this function? edtior stabilization sounds a bit unclear to me, maybe something like waitForTextGeneration is better?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea!

const startTime = Date.now()
let previousLines = await editor.getNumberOfLines()
let stableCount = 0

while (Date.now() - startTime < timeout) {
await sleep(1000)
const currentLines = await editor.getNumberOfLines()

if (currentLines === previousLines) {
stableCount++
if (stableCount >= 2) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whats the logic for breaking when stableCount is >= 2? does that mean that as soon as we see more then 2 lines we break?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, this essentially will check for if the number of lines has not changed yet. If this number of lines being the same occurs twice (requiring another sleep), then we can say that the model has stopped generating code or thinking. Although it might be more useful to say this is just greater than 2, rather than equal to 2.

return
}
} else {
stableCount = 0
}

previousLines = currentLines
}

throw new Error(`Editor stabilization timed out after ${timeout}ms`)
}

/**
Expand Down