Skip to content

Commit c6a13a8

Browse files
authored
test(webview): skip flaky test on min vscode #5955
## Problem It appears that older versions of VSCode are causing this test to be flaky, which appears to be fixed in v1.88. See findings section for more information. ## Solution - Skip the test if vscode version is pre-1.88. - Add tech-debt test to remove this check once min is 1.88+. ## Findings - Ran 1000 times in CI, and failed 29 times. (2.9% failure rate). If we re-reran 3 times, ci would pass ~ 99.998% of the time (assuming failures independent). - Only fails on VSCode minimum (1.83.0). - Originally thought it was related to activating Q extension or fetching it from CDN, but appears mocking this still results in flakiness. - Every failure comes with a `ERROR:gles2_command_buffer_stub.cc(364)] ContextResult::kFatalFailure Failed to create context.` which appears to be from OpenGL. This means it is coming from very far down the stack since VSCode is built on electron, which uses chromium, which is then using OpenGL. (ai suggests this is a GPU problem?). - Thought to be from race condition with async constructor hack, fixed by #5961. - Evidence is suggesting that this could be a bug in a prior version of VSCode, electron, or OpenGL, and should go away once we bump minimum VSCode version. - Works on VSCode 1.90+, 1.87- fails, 1.88 is the minimum version it is not flaky. (verified using 1000 attempts). - Failures are NOT independent since retrying 10x times only reduced failure rate to ~1%. Therefore retries are not a viable option.
1 parent 10b3f1c commit c6a13a8

File tree

1 file changed

+6
-5
lines changed

1 file changed

+6
-5
lines changed

packages/core/src/test/applicationcomposer/messageHandlers/generateResourceHandler.test.ts

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@ import sinon from 'sinon'
88
import { createTemplate, createWebviewContext } from '../utils'
99
import { generateResourceHandler } from '../../../applicationcomposer/messageHandlers/generateResourceHandler'
1010
import { Command, MessageType } from '../../../applicationcomposer/types'
11+
import * as env from '../../../shared/vscode/env'
1112

1213
describe('generateResourceHandler', function () {
13-
afterEach(() => {
14-
sinon.restore()
15-
})
16-
17-
it('amazon q is not installed', async () => {
14+
it('amazon q is not installed', async function () {
15+
if (env.isMinVscode('1.89.0')) {
16+
this.skip()
17+
}
1818
const panel = await createTemplate()
1919
const postMessageSpy = sinon.spy(panel.webview, 'postMessage')
2020
const context = await createWebviewContext({
@@ -32,5 +32,6 @@ describe('generateResourceHandler', function () {
3232
)
3333
assert.ok(postMessageSpy.calledOnce)
3434
assert.deepStrictEqual(postMessageSpy.getCall(0).args[0].isSuccess, false)
35+
postMessageSpy.restore()
3536
})
3637
})

0 commit comments

Comments
 (0)