-
Notifications
You must be signed in to change notification settings - Fork 274
test(qdoc): Add ui tests for qdoc update readme flow #5435
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 15 commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
d142bd3
test(qdoc): initial tests for latest code changes flow
62d6bff
update tstData to test
43b4617
update scripts + move test data to subfolder for update readme tests
a004a28
update instructions in update readme with specific changes test
53ceb04
update base readme for testing the update readme flow
22809cc
update test assertions for q doc update flow tests
c00d3cd
separate out test cases into individual tests
525d784
remove gradle settings file for q doc update flow test repo
fa120b2
log result from scripts for debugging
f1fa2c8
manually restore readme after each test run
c4a34f5
add end of assertions println for debugging + update readme
e6b30c0
update q tests config for debugging
b730f6d
update config and rename dummy repo files for create readme tests
deb3d94
update build file for create readme tests sample repo
7264ff0
update to print the result on failures only
de4afb2
remove unused file from test dummy repo
93ede19
Merge branch 'aws:main' into doc
gandhi-21 50923b7
fix detekt issues
1d1b5a2
Merge branch 'main' into doc
gandhi-21 0ca4932
fix detekt issues for package declaration
7f9f881
move tests cases into single test class for update readme tests
e99c056
move envSetup to BeforeAll and testDataSetup to BeforeEach
cc99a6c
fix detekt issues
e828a95
fix detekt issues
28a1045
Merge branch 'main' into doc
gandhi-21 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
172 changes: 172 additions & 0 deletions
172
...e/aws/toolkits/jetbrains/uitests/docTests/scripts/UpdateReadmeWithLatestChangesScripts.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,172 @@ | ||
| // Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package software.aws.toolkits.jetbrains.uitests.docTests.scripts | ||
|
|
||
| // language=TS | ||
| val updateReadmeLatestChangesConfirmOptionsScript = """ | ||
| const puppeteer = require('puppeteer'); | ||
|
|
||
| async function testNavigation() { | ||
| const browser = await puppeteer.connect({ | ||
| browserURL: 'http://localhost:9222' | ||
| }) | ||
|
|
||
| try { | ||
|
|
||
| const pages = await browser.pages() | ||
|
|
||
| for(const page of pages) { | ||
| const contents = await page.evaluate(el => el.innerHTML, await page.${'$'}(':root')); | ||
|
|
||
| const element = await page.${'$'}('.mynah-chat-prompt-input') | ||
| if(element) { | ||
|
|
||
| console.log('Typing /doc in the chat window') | ||
|
|
||
| await page.type('.mynah-chat-prompt-input', '/doc') | ||
| await page.keyboard.press('Enter') | ||
|
|
||
| console.log('Attempting to find and click Update an existing README button') | ||
| await findAndClickButton(page, 'Update an existing README', true, 10000) | ||
| console.log('Attempting to find and click Update README to reflect code button') | ||
| await findAndClickButton(page, 'Update README to reflect code', true, 10000) | ||
| console.log('Attempting to find all available buttons') | ||
| const yesButton = await findAndClickButton(page, 'Yes', false, 10000) | ||
| const changeFolderButton = await findAndClickButton(page, 'Change folder', false, 10000) | ||
| const cancelButton = await findAndClickButton(page, 'Cancel', false, 10000) | ||
|
|
||
| if (!yesButton || !changeFolderButton || !cancelButton) { | ||
| console.log('Error: Test Failed') | ||
| console.log('Unable to find buttons for Yes/ChangeFolder/Cancel') | ||
| } else { | ||
| console.log('Found all expected buttons') | ||
| console.log('Test Successful') | ||
| } | ||
| } | ||
| } | ||
|
|
||
| } finally { | ||
| await browser.close(); | ||
| } | ||
| } | ||
|
|
||
| testNavigation().catch((error) => { | ||
| console.log('Error: Test Failed'); | ||
| console.error(error); | ||
| }); | ||
| """.trimIndent() | ||
|
|
||
| // language=TS | ||
| val updateReadmeLatestChangesScript = """ | ||
|
|
||
| const puppeteer = require('puppeteer'); | ||
|
|
||
| async function testNavigation() { | ||
| const browser = await puppeteer.connect({ | ||
| browserURL: 'http://localhost:9222' | ||
| }) | ||
|
|
||
| try { | ||
|
|
||
| const pages = await browser.pages() | ||
|
|
||
| for(const page of pages) { | ||
| const contents = await page.evaluate(el => el.innerHTML, await page.${'$'}(':root')); | ||
|
|
||
| const element = await page.${'$'}('.mynah-chat-prompt-input') | ||
|
|
||
| if(element) { | ||
|
|
||
| console.log('Typing /doc in the chat window') | ||
|
|
||
| await page.type('.mynah-chat-prompt-input', '/doc') | ||
| await page.keyboard.press('Enter') | ||
|
|
||
| console.log('Attempting to find and click Update an existing README button') | ||
| await findAndClickButton(page, 'Update an existing README', true, 10000) | ||
| console.log('Attempting to find and click Update README to reflect code button') | ||
| await findAndClickButton(page, 'Update README to reflect code', true, 10000) | ||
| console.log('Attempting to find and click Yes button to confirm option') | ||
| await findAndClickButton(page, 'Yes', true, 10000) | ||
| console.log('Waiting for updated README to be generated') | ||
| await new Promise(resolve => setTimeout(resolve, 90000)); | ||
| console.log('Attempting to find and click Accept button') | ||
| await findAndClickButton(page, 'Accept', true, 10000) | ||
| } | ||
| } | ||
|
|
||
| } finally { | ||
| await browser.close(); | ||
| } | ||
| } | ||
|
|
||
| testNavigation().catch((error) => { | ||
| console.log('Error: Test Failed'); | ||
| console.error(error); | ||
| }); | ||
|
|
||
| """.trimIndent() | ||
|
|
||
| // language=TS | ||
| val updateReadmeLatestChangesMakeChangesFlowScript = """ | ||
|
|
||
| const puppeteer = require('puppeteer'); | ||
|
|
||
| async function testNavigation() { | ||
| const browser = await puppeteer.connect({ | ||
| browserURL: 'http://localhost:9222' | ||
| }); | ||
|
|
||
| try { | ||
|
|
||
| const pages = await browser.pages(); | ||
|
|
||
| for(const page of pages) { | ||
| const contents = await page.evaluate(el => el.innerHTML, await page.${'$'}(':root')); | ||
|
|
||
| const element = await page.${'$'}('.mynah-chat-prompt-input'); | ||
| if(element) { | ||
|
|
||
| console.log('Typing /doc in the chat window'); | ||
|
|
||
| await page.type('.mynah-chat-prompt-input', '/doc'); | ||
| await page.keyboard.press('Enter'); | ||
|
|
||
| console.log('Attempting to find and click Update an existing README button'); | ||
| await findAndClickButton(page, 'Update an existing README', true, 10000); | ||
| console.log('Attempting to find and click Update README to reflect code button'); | ||
| await findAndClickButton(page, 'Update README to reflect code', true, 10000); | ||
| console.log('Attempting to find and click Yes button to confirm option'); | ||
| await findAndClickButton(page, 'Yes', true, 10000); | ||
| console.log('Waiting for updated README to be generated'); | ||
| await new Promise(resolve => setTimeout(resolve, 90000)); | ||
| console.log('Attempting to find and click Make changes button'); | ||
| await findAndClickButton(page, 'Make changes', true, 10000); | ||
| const makeChangeText = await page.waitForSelector('[placeholder="Describe documentation changes"]'); | ||
| if (!makeChangeText) { | ||
| console.log('Error: Test Failed'); | ||
| console.log('Unable to find placeholder description test in Make Changes flow'); | ||
| } else { | ||
| console.log('Found expected placeholder text for Make Changes flow'); | ||
| console.log('Test Successful'); | ||
| } | ||
|
|
||
| } | ||
| } | ||
|
|
||
| } finally { | ||
| await browser.close(); | ||
| } | ||
| } | ||
|
|
||
| testNavigation().catch((error) => { | ||
| console.log('Error: Test Failed'); | ||
| console.error(error); | ||
| }); | ||
|
|
||
| """.trimIndent() | ||
|
|
||
| val updateReadmeLatestChangesConfirmOptionsTestScript = updateReadmeLatestChangesConfirmOptionsScript.plus(findAndClickButtonScript) | ||
| val updateReadmeLatestChangesTestScript = updateReadmeLatestChangesScript.plus(findAndClickButtonScript) | ||
| val updateReadmeLatestChangesMakeChangesFlowTestScript = updateReadmeLatestChangesMakeChangesFlowScript.plus(findAndClickButtonScript) | ||
128 changes: 128 additions & 0 deletions
128
...aws/toolkits/jetbrains/uitests/docTests/scripts/UpdateReadmeWithSpecificChangesScripts.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,128 @@ | ||
| // Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package software.aws.toolkits.jetbrains.uitests.docTests.scripts | ||
|
|
||
| // language=TS | ||
| val updateReadmeSpecificChangesMakeChangesFlowScript = """ | ||
|
|
||
| const puppeteer = require('puppeteer'); | ||
|
|
||
| async function testNavigation() { | ||
| const browser = await puppeteer.connect({ | ||
| browserURL: 'http://localhost:9222' | ||
| }); | ||
|
|
||
| try { | ||
|
|
||
| const pages = await browser.pages(); | ||
|
|
||
| for(const page of pages) { | ||
| const contents = await page.evaluate(el => el.innerHTML, await page.${'$'}(':root')); | ||
|
|
||
| const element = await page.${'$'}('.mynah-chat-prompt-input'); | ||
| if(element) { | ||
|
|
||
| console.log('Typing /doc in the chat window'); | ||
|
|
||
| await page.type('.mynah-chat-prompt-input', '/doc'); | ||
| await page.keyboard.press('Enter'); | ||
|
|
||
| console.log('Attempting to find and click Update an existing README button'); | ||
| await findAndClickButton(page, 'Update an existing README', true, 10000); | ||
| console.log('Attempting to find and click Make a specific change button'); | ||
| await findAndClickButton(page, 'Make a specific change', true, 10000); | ||
| console.log('Attempting to find and click Yes button to confirm option'); | ||
| await findAndClickButton(page, 'Yes', true, 10000); | ||
|
|
||
| console.log('Typing specific change instructions in the chat window'); | ||
| await page.type('.mynah-chat-prompt-input', 'Add a section with Installation instructions for this repository. Title this new section \"### Installation\"'); | ||
| await page.keyboard.press('Enter'); | ||
|
|
||
| console.log('Waiting for updated README to be generated'); | ||
| await new Promise(resolve => setTimeout(resolve, 90000)); | ||
| console.log('Attempting to find and click Make changes button'); | ||
| await findAndClickButton(page, 'Make changes', true, 10000); | ||
| const makeChangeText = await page.waitForSelector('[placeholder="Describe documentation changes"]'); | ||
| if (!makeChangeText) { | ||
| console.log('Error: Test Failed'); | ||
| console.log('Unable to find placeholder description text in Make Changes flow'); | ||
| } else { | ||
| console.log('Found expected placeholder text for Make Changes flow'); | ||
| console.log('Test Successful'); | ||
| } | ||
|
|
||
| } | ||
| } | ||
|
|
||
| } finally { | ||
| await browser.close(); | ||
| } | ||
| } | ||
|
|
||
| testNavigation().catch((error) => { | ||
| console.log('Error: Test Failed'); | ||
| console.error(error); | ||
| }); | ||
|
|
||
| """.trimIndent() | ||
|
|
||
| // language=TS | ||
| val updateReadmeSpecificChangesScript = """ | ||
|
|
||
| const puppeteer = require('puppeteer'); | ||
|
|
||
| async function testNavigation() { | ||
| const browser = await puppeteer.connect({ | ||
| browserURL: 'http://localhost:9222' | ||
| }); | ||
|
|
||
| try { | ||
|
|
||
| const pages = await browser.pages(); | ||
|
|
||
| for(const page of pages) { | ||
| const contents = await page.evaluate(el => el.innerHTML, await page.${'$'}(':root')); | ||
|
|
||
| const element = await page.${'$'}('.mynah-chat-prompt-input'); | ||
|
|
||
| if(element) { | ||
|
|
||
| console.log('Typing /doc in the chat window'); | ||
|
|
||
| await page.type('.mynah-chat-prompt-input', '/doc'); | ||
| await page.keyboard.press('Enter'); | ||
|
|
||
| console.log('Attempting to find and click Update an existing README button'); | ||
| await findAndClickButton(page, 'Update an existing README', true, 10000); | ||
| console.log('Attempting to find and click Make a specific change button'); | ||
| await findAndClickButton(page, 'Make a specific change', true, 10000); | ||
| console.log('Attempting to find and click Yes button to confirm option'); | ||
| await findAndClickButton(page, 'Yes', true, 10000); | ||
|
|
||
| console.log('Typing specific change instructions in the chat window'); | ||
| await page.type('.mynah-chat-prompt-input', 'Add a section with Installation instructions for this repository. Title this new section \"### Installation\"'); | ||
| await page.keyboard.press('Enter'); | ||
|
|
||
| console.log('Waiting for updated README to be generated'); | ||
| await new Promise(resolve => setTimeout(resolve, 90000)); | ||
|
|
||
| console.log('Attempting to find and click Accept button'); | ||
| await findAndClickButton(page, 'Accept', true, 10000); | ||
| } | ||
| } | ||
|
|
||
| } finally { | ||
| await browser.close(); | ||
| } | ||
| } | ||
|
|
||
| testNavigation().catch((error) => { | ||
| console.log('Error: Test Failed'); | ||
| console.error(error); | ||
| }); | ||
|
|
||
| """.trimIndent() | ||
|
|
||
| val updateReadmeSpecificChangesMakeChangesFlowTestScript = updateReadmeSpecificChangesMakeChangesFlowScript.plus(findAndClickButtonScript) | ||
gandhi-21 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| val updateReadmeSpecificChangesTestScript = updateReadmeSpecificChangesScript.plus(findAndClickButtonScript) | ||
71 changes: 71 additions & 0 deletions
71
ui-tests-starter/tst-243+/software/aws/toolkits/jetbrains/uitests/docTests/scripts/Utils.kt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| // Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package software.aws.toolkits.jetbrains.uitests.docTests.scripts | ||
|
|
||
| // language=TS | ||
| val findAndClickButtonScript = """ | ||
| const findAndClickButton = async ( | ||
| page, | ||
| buttonText, | ||
| clickButton = false, | ||
| timeout = 5000 | ||
| ) => { | ||
| try { | ||
| // Wait for any matching buttons to be present | ||
| await page.waitForSelector('button.mynah-button', { | ||
| visible: true, | ||
| timeout, | ||
| }); | ||
|
|
||
| // Find and verify the specific button | ||
| const buttonHandle = await page.evaluateHandle(text => { | ||
| const buttons = Array.from( | ||
| document.querySelectorAll('button.mynah-button') | ||
| ); | ||
| return buttons.find(button => { | ||
| const label = button.querySelector('.mynah-button-label'); | ||
| return label && label.textContent.trim() === text; | ||
| }); | ||
| }, buttonText); | ||
|
|
||
| // Check if button was found | ||
| const button = buttonHandle.asElement(); | ||
| if (!button) { | ||
| console.log(buttonText); | ||
| throw new Error(`Button with text not found`); | ||
| } | ||
|
|
||
| // Verify button is visible and enabled | ||
| const isVisible = await page.evaluate(el => { | ||
| const style = window.getComputedStyle(el); | ||
| return ( | ||
| style.display !== 'none' && | ||
| style.visibility !== 'hidden' && | ||
| style.opacity !== '0' | ||
| ); | ||
| }, button); | ||
|
|
||
| if (!isVisible) { | ||
| console.log(buttonText); | ||
| throw new Error(`Button with text is not visible`); | ||
| } | ||
|
|
||
| if (clickButton) { | ||
| // Click the button | ||
| await button.click(); | ||
|
|
||
| // Optional wait after click | ||
| await new Promise(resolve => setTimeout(resolve, 1000)); | ||
|
|
||
| console.log(`Successfully clicked button with text`); | ||
| console.log(buttonText); | ||
| } else { | ||
| return button; | ||
| } | ||
| } catch (error) { | ||
| console.error(`Error interacting with button:`, buttonText, error); | ||
| throw error; | ||
| } | ||
| }; | ||
| """.trimIndent() |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These few lines appear to be repeated in most tests, can they be abstracted?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
abstracting parts of the testing flow can cause issues while trying to read and debug code for someone unfamiliar. So we should keep testing scripts as is to understand what's happening in the test without needing to jump around.