Skip to content

Commit c76cded

Browse files
test(lambda): fix multiple flaky tests (#8142)
## Problem 3 flaky appbuilder tests ``` 2 failing 1) AppBuilder Walkthrough Create project start with an existing template override existing template download serverlessland proj: AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected + 'Encountered an unexpected error when processing the request, please try again' - 'template.yaml already exist in the selected directory, overwrite?' + expected - actual Error: Test length exceeded max duration: 30 seconds [No Pending UI Elements Found] -Encountered an unexpected error when processing the request, please try again +template.yaml already exist in the selected directory, overwrite? at /Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/test/awsService/appBuilder/walkthrough.test.ts:196:32 at s.value (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/test/shared/vscode/window.ts:502:35) at Proxy.z (/private/tmp/.vscode-test/vscode-darwin-arm64-1.83.0/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:80:1902) at Proxy.fire (/private/tmp/.vscode-test/vscode-darwin-arm64-1.83.0/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/api/node/extensionHostProcess.js:80:2119) at fireOnDidShowMessage (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/test/shared/vscode/window.ts:147:35) at /Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/test/shared/vscode/message.ts:247:27 at new Promise (<anonymous>) at Proxy.<anonymous> (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/test/shared/vscode/message.ts:244:20) at errorPromptHelper (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/commands/startSecurityScan.ts:485:28) at startSecurityScan (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/codewhisperer/commands/startSecurityScan.ts:363:13) 2) SamDebugConfigurationProvider makeConfig failure modes: AssertionError [ERR_ASSERTION]: Missing expected rejection. at async Context.<anonymous> (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/test/shared/sam/debugger/samDebugConfigProvider.test.ts:320:13) 1) SAM runBuild :) path [entry: command palette] use samconfig should instantiate correct process in terminal: AssertionError [ERR_ASSERTION]: RunInterminal 1 !== 2 + expected - actual -1 +2 at verifyCorrectDependencyCall (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/test/shared/sam/build.test.ts:344:20) at Context.<anonymous> (/Users/runner/work/aws-toolkit-vscode/aws-toolkit-vscode/packages/core/src/test/shared/sam/build.test.ts:532:13) https://d1ihu6zq92vp9p.cloudfront.net/b4a5497f-734f-491d-a40e-795359615103/report.html ``` https://d1ihu6zq92vp9p.cloudfront.net/2d3ec18e-1584-4a40-948c-18babdfee24e/report.html 3 integ test constantly failing, 2 due to node 18 deprecated, 1 due to an uncaught change before ``` 1) SAM Integration Tests SAM runtime: nodejs18.x (ZIP) Starting with a newly created nodejs18.x (ZIP) SAM Application... "before all" hook for "produces an Add Debug Configuration codelens": CodeExpectedError: cannot open file:///codebuild/output/src2664791430/src/github.com/aws/aws-toolkit-vscode/packages/core/dist/src/testFixtures/workspaceFolder/inttestco5XDH/nodejs18.x/samapp-WKxmw3/testProject/hello-world/app.mjs. Detail: Unable to read file '/codebuild/output/src2664791430/src/github.com/aws/aws-toolkit-vscode/packages/core/dist/src/testFixtures/workspaceFolder/inttestco5XDH/nodejs18.x/samapp-WKxmw3/testProject/hello-world/app.mjs' (Error: Unable to resolve nonexistent file '/codebuild/output/src2664791430/src/github.com/aws/aws-toolkit-vscode/packages/core/dist/src/testFixtures/workspaceFolder/inttestco5XDH/nodejs18.x/samapp-WKxmw3/testProject/hello-world/app.mjs') at s.$tryOpenDocument (vscode-file://vscode-app/tmp/.vscode-test/vscode-linux-x64-1.83.0/resources/app/out/vs/workbench/workbench.desktop.main.js:2343:37719) 2) SAM Integration Tests SAM runtime: nodejs18.x (Image) Starting with a newly created nodejs18.x (Image) SAM Application... "before all" hook for "produces an Add Debug Configuration codelens": SamCliError: Please verify your location. The following types of location are supported: [SamCliFailed] at makeUnexpectedExitCodeError (/codebuild/output/src2664791430/src/github.com/aws/aws-toolkit-vscode/packages/core/src/shared/sam/cli/samCliInvokerUtils.ts:48:12) at logAndThrowIfUnexpectedExitCode (/codebuild/output/src2664791430/src/github.com/aws/aws-toolkit-vscode/packages/core/src/shared/sam/cli/samCliInvokerUtils.ts:67:11) at runSamCliInit (/codebuild/output/src2664791430/src/github.com/aws/aws-toolkit-vscode/packages/core/src/shared/sam/cli/samCliInit.ts:65:36) at async createSamApplication (/codebuild/output/src2664791430/src/github.com/aws/aws-toolkit-vscode/packages/core/src/testInteg/sam.test.ts:640:9) at async Context.<anonymous> (/codebuild/output/src2664791430/src/github.com/aws/aws-toolkit-vscode/packages/core/src/testInteg/sam.test.ts:382:21) 3) Application Builder application nodes in workspace (Test in order) 3: contains correct resource node properties: AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: 1 !== 0 + expected - actual -1 +0 at Context.<anonymous> (/codebuild/output/src2664791430/src/github.com/aws/aws-toolkit-vscode/packages/core/src/testInteg/appBuilder/sidebar/appBuilderNode.test.ts:132:20) at processTicksAndRejections (node:internal/process/task_queues:95:5) ``` ## Solution - make sure codewhisper doesn't show error in unrelated test - ensure clean state in sam tests - remove deprecated node 18 test cases --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license. Co-authored-by: Laxman Reddy <[email protected]>
1 parent 9453746 commit c76cded

File tree

5 files changed

+16
-7
lines changed

5 files changed

+16
-7
lines changed

packages/core/src/codewhisperer/commands/startSecurityScan.ts

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -482,7 +482,10 @@ export function errorPromptHelper(
482482
})
483483
}
484484
if (error.code !== 'NoSourceFilesError') {
485-
void vscode.window.showWarningMessage(getErrorMessage(error), ok)
485+
// Skip showing warning messages during tests to avoid interfering with test dialogs
486+
if (process.env.NODE_ENV !== 'test') {
487+
void vscode.window.showWarningMessage(getErrorMessage(error), ok)
488+
}
486489
}
487490
}
488491

packages/core/src/test/shared/sam/build.test.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -512,6 +512,9 @@ describe('SAM runBuild', () => {
512512
})
513513
.build()
514514

515+
// Reset the spy before running the test to ensure clean state
516+
spyRunInterminal.resetHistory()
517+
515518
// Instead of await runBuild(), prefer this to avoid flakiness due to race condition
516519
await delayedRunBuild()
517520

packages/core/src/test/shared/sam/debugger/samDebugConfigProvider.test.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -317,7 +317,12 @@ describe('SamDebugConfigurationProvider', async function () {
317317
)
318318

319319
// No workspace folder:
320+
// Stub vscode.workspace.workspaceFolders to be undefined to ensure rejection
321+
sandbox.stub(vscode.workspace, 'workspaceFolders').value(undefined)
320322
await assert.rejects(() => debugConfigProvider.makeConfig(undefined, config.config))
323+
// Restore for subsequent tests
324+
sandbox.restore()
325+
sandbox = sinon.createSandbox()
321326

322327
// No launch.json (vscode will pass an empty config.request):
323328
await assert.rejects(() => debugConfigProvider.makeConfig(undefined, { ...config.config, request: '' }))

packages/core/src/testInteg/appBuilder/sidebar/appBuilderNode.test.ts

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -129,28 +129,28 @@ describe('Application Builder', async () => {
129129
)
130130
assert.strictEqual(lambdaResourceNode.id, 'AppBuilderProjectLambda')
131131
const lambdaTreeItemProperties = lambdaResourceNode.getTreeItem()
132-
assert.strictEqual(lambdaTreeItemProperties.collapsibleState, vscode.TreeItemCollapsibleState.None)
132+
assert.strictEqual(lambdaTreeItemProperties.collapsibleState, vscode.TreeItemCollapsibleState.Collapsed)
133133
assert.strictEqual(lambdaTreeItemProperties.iconPath?.toString(), '$(aws-lambda-function)')
134134

135135
// Validate s3 bucket
136136
const s3BucketResourceNode = getResourceNodeByType(appBuilderTestAppResourceNodes, 'AWS::S3::Bucket')
137137
assert.strictEqual(s3BucketResourceNode.id, 'AppBuilderProjectBucket')
138138
const s3BucketTreeItemProperties = s3BucketResourceNode.getTreeItem()
139-
assert.strictEqual(s3BucketTreeItemProperties.collapsibleState, vscode.TreeItemCollapsibleState.None)
139+
assert.strictEqual(s3BucketTreeItemProperties.collapsibleState, vscode.TreeItemCollapsibleState.Collapsed)
140140
assert.strictEqual(s3BucketTreeItemProperties.iconPath?.toString(), '$(aws-s3-bucket)')
141141

142142
// Validate s3 policy
143143
const s3PolicyResourceNode = getResourceNodeByType(appBuilderTestAppResourceNodes, 'AWS::S3::BucketPolicy')
144144
assert.strictEqual(s3PolicyResourceNode.id, 'AppBuilderProjectBucketBucketPolicy')
145145
const s3PolicyTreeItemProperties = s3PolicyResourceNode.getTreeItem()
146-
assert.strictEqual(s3PolicyTreeItemProperties.collapsibleState, vscode.TreeItemCollapsibleState.None)
146+
assert.strictEqual(s3PolicyTreeItemProperties.collapsibleState, vscode.TreeItemCollapsibleState.Collapsed)
147147
assert.strictEqual(s3PolicyTreeItemProperties.iconPath?.toString(), '$(info)')
148148

149149
// Validate api gateway resource node
150150
const apigwResourceNode = getResourceNodeByType(appBuilderTestAppResourceNodes, 'AWS::Serverless::Api')
151151
assert.strictEqual(apigwResourceNode.id, 'AppBuilderProjectAPI')
152152
const apigwTreeItemProperties = apigwResourceNode.getTreeItem()
153-
assert.strictEqual(apigwTreeItemProperties.collapsibleState, vscode.TreeItemCollapsibleState.None)
153+
assert.strictEqual(apigwTreeItemProperties.collapsibleState, vscode.TreeItemCollapsibleState.Collapsed)
154154
assert.strictEqual(apigwTreeItemProperties.iconPath?.toString(), '$(info)')
155155
})
156156

packages/core/src/testInteg/sam.test.ts

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,6 @@ function generateScenario(
123123

124124
const scenarios: TestScenario[] = [
125125
// zips
126-
generateScenario('nodejs', '18.x'),
127126
generateScenario('nodejs', '20.x'),
128127
generateScenario('nodejs', '22.x', { vscodeMinimum: '1.78.0' }),
129128
generateScenario('python', '3.10'),
@@ -135,7 +134,6 @@ const scenarios: TestScenario[] = [
135134
generateScenario('java', '11', { sourceTag: 'Gradle' }),
136135
generateScenario('java', '17', { sourceTag: 'Gradle' }),
137136
// images
138-
generateScenario('nodejs', '18.x', { baseImage: 'amazon/nodejs18.x-base' }, true),
139137
generateScenario('nodejs', '20.x', { baseImage: 'amazon/nodejs20.x-base' }, true),
140138
generateScenario('nodejs', '22.x', { baseImage: 'amazon/nodejs22.x-base', vscodeMinimum: '1.78.0' }, true),
141139
generateScenario('python', '3.10', { baseImage: 'amazon/python3.10-base' }, true),

0 commit comments

Comments
 (0)