Skip to content

Commit 7991855

Browse files
cpcallencpcallen
authored
test: Make tests faster, more robust, more consistent (#692)
* fix(tests): Restore window size after scroll tests * docs(tests): "Selenium" -> "WebdriverIO" We aren't using Selenium, so we shouldn't be saying we do. * chore(tests): Make timeout(0) calls only when PAUSE_TIME is nonzero With PAUSE_TIME set to zero, on my relatively fast laptop none of these tests run slow enough to fail due to the default 2s timeous, and even with PAUSE_TIME set to 50ms the slowest test is 1024ms (of which 700ms is spent in brower.pause calls inside sendKeyAndWait). We do want to disable timeouts when debugging tests, both because we might set PAUSE_TIME to a larger value and because we might use the debugger to pause things entirely, but we don't need to disable timeouts when running tests just to check their results. * fix(tests): Set PAUSE_TIME to 0 It's very useful to be able to make tests run more slowly so you can watch them, but they should run as fast as possible by default. This cuts total test execution time on a 2021 MacBook Pro M1 approximately in half, from 42s to 22s. * refactor(tests): Use sendKeyAndWait where appropriate In most places we were already following browser.keys with a browser.pause(PAUSE_TIME), so using sendKeysAndWait is more succinct; in other places we didn't have the pause but it is not harmful to add a pause (especially now the default PAUSE_TIME is zero) and could be helpful when watching tests run with a non-zero PAUSE_TIME. * fix(tests): Make tabNavigateToWorkspace idempotent Previously this function would just send a bunch of tabs, which depended on focus state being as-on-document-load. Some tests (notably the ones in basic_test.ts) that have only a suiteSetup and not a (per-test) setup method were only were only passing because of the combination of: * Due to issue #632, pressing tab when the workspace is focused (and there are no further focusable elements on the page) incorrectly causes focus to move to the first focusable element on the page instead of (as would normally be the case) to the browser controls, and * The fact that the index.html had exactly one additional focusable div on the page, preceding the injection div. This meant that calling tabNavigateToWorkspace when the workspace was already focused would, only by sheer coincidence, result in the focus remaining on the workspace. By explicitly focusing a known element, tabNavigateToWorkspace should work correctly regardless of how many focusable elements are on the page and which one (if any) was focused before the call. * chore(tests): Remove unneeded tabNavigateToWorkspace calls Any time a tabNavigateToWorkspace call is followed by a call to focusOnBlock the former can be removed with no discernable effect except to make tests run slightly faster and with less flashing of the flyout. * fix(tests): Increase timeout for certain slow tests * docs(tests): Fix typo * chore(tests): Add missing timeout(0) calls (when PAUSE_TIME is nonzero) These two files were inadvertently omitted from commit 14d619c. * fix(tests): Lint * fix(tests): Add missing import
1 parent e16f5aa commit 7991855

17 files changed

+203
-241
lines changed

‎test/webdriverio/test/actions_test.ts‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -14,27 +14,31 @@ import {
1414
tabNavigateToWorkspace,
1515
testFileLocations,
1616
testSetup,
17+
sendKeyAndWait,
1718
keyRight,
1819
contextMenuItems,
1920
} from './test_setup.js';
2021

2122
suite('Menus test', function () {
22-
// Setting timeout to unlimited as these tests take longer time to run
23-
this.timeout(0);
23+
// Disable timeouts when non-zero PAUSE_TIME is used to watch tests run.
24+
if (PAUSE_TIME) this.timeout(0);
2425

25-
// Clear the workspace and load start blocks
26+
// Clear the workspace and load start blocks.
2627
setup(async function () {
28+
// This is the first test suite, which must wait for Chrome +
29+
// chromedriver to start up, which can be slow—perhaps a few
30+
// seconds. Allow 30s just in case.
31+
this.timeout(30000);
32+
2733
this.browser = await testSetup(testFileLocations.BASE);
2834
await this.browser.pause(PAUSE_TIME);
2935
});
3036

3137
test('Menu on block', async function () {
3238
// Navigate to draw_circle_1.
33-
await tabNavigateToWorkspace(this.browser);
3439
await focusOnBlock(this.browser, 'draw_circle_1');
3540
await this.browser.pause(PAUSE_TIME);
36-
await this.browser.keys([Key.Ctrl, Key.Return]);
37-
await this.browser.pause(PAUSE_TIME);
41+
await sendKeyAndWait(this.browser, [Key.Ctrl, Key.Return]);
3842

3943
chai.assert.deepEqual(
4044
process.platform === 'darwin'
@@ -70,14 +74,12 @@ suite('Menus test', function () {
7074

7175
test('Menu on block in the toolbox', async function () {
7276
// Navigate to draw_circle_1.
73-
await tabNavigateToWorkspace(this.browser);
7477
await focusOnBlock(this.browser, 'draw_circle_1');
7578
// Navigate to a toolbox category
7679
await moveToToolboxCategory(this.browser, 'Functions');
7780
// Move to flyout.
7881
await keyRight(this.browser);
79-
await this.browser.keys([Key.Ctrl, Key.Return]);
80-
await this.browser.pause(PAUSE_TIME);
82+
await sendKeyAndWait(this.browser, [Key.Ctrl, Key.Return]);
8183

8284
chai.assert.deepEqual(
8385
process.platform === 'darwin'
@@ -100,9 +102,8 @@ suite('Menus test', function () {
100102
test('Menu on workspace', async function () {
101103
// Navigate to draw_circle_1.
102104
await tabNavigateToWorkspace(this.browser);
103-
await this.browser.keys('w');
104-
await this.browser.keys([Key.Ctrl, Key.Return]);
105-
await this.browser.pause(PAUSE_TIME);
105+
await sendKeyAndWait(this.browser, 'w');
106+
await sendKeyAndWait(this.browser, [Key.Ctrl, Key.Return]);
106107

107108
chai.assert.deepEqual(
108109
process.platform === 'darwin'
@@ -132,12 +133,11 @@ suite('Menus test', function () {
132133

133134
test('Menu on block during drag is not shown', async function () {
134135
// Navigate to draw_circle_1.
135-
await tabNavigateToWorkspace(this.browser);
136136
await focusOnBlock(this.browser, 'draw_circle_1');
137137
// Start moving the block
138-
await this.browser.keys('m');
139-
await this.browser.keys([Key.Ctrl, Key.Return]);
140-
await this.browser.pause(PAUSE_TIME);
138+
await sendKeyAndWait(this.browser, 'm');
139+
await sendKeyAndWait(this.browser, [Key.Ctrl, Key.Return]);
140+
141141
chai.assert.isTrue(
142142
await contextMenuExists(this.browser, 'Collapse Block', true),
143143
'The menu should not be openable during a move',

‎test/webdriverio/test/basic_test.ts‎

Lines changed: 12 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import {
1616
testSetup,
1717
testFileLocations,
1818
PAUSE_TIME,
19+
sendKeyAndWait,
1920
tabNavigateToWorkspace,
2021
keyLeft,
2122
keyRight,
@@ -25,10 +26,10 @@ import {
2526
import {Key} from 'webdriverio';
2627

2728
suite('Keyboard navigation on Blocks', function () {
28-
// Setting timeout to unlimited as these tests take a longer time to run than most mocha test
29-
this.timeout(0);
29+
// Disable timeouts when non-zero PAUSE_TIME is used to watch tests run.
30+
if (PAUSE_TIME) this.timeout(0);
3031

31-
// Setup Selenium for all of the tests
32+
// Clear the workspace and load start blocks.
3233
suiteSetup(async function () {
3334
this.browser = await testSetup(testFileLocations.NAVIGATION_TEST_BLOCKS);
3435
});
@@ -53,8 +54,6 @@ suite('Keyboard navigation on Blocks', function () {
5354
});
5455

5556
test('Down from statement block selects next block across stacks', async function () {
56-
await tabNavigateToWorkspace(this.browser);
57-
await this.browser.pause(PAUSE_TIME);
5857
await focusOnBlock(this.browser, 'p5_canvas_1');
5958
await this.browser.pause(PAUSE_TIME);
6059
await keyDown(this.browser);
@@ -65,8 +64,6 @@ suite('Keyboard navigation on Blocks', function () {
6564
});
6665

6766
test('Up from statement block selects previous block', async function () {
68-
await tabNavigateToWorkspace(this.browser);
69-
await this.browser.pause(PAUSE_TIME);
7067
await focusOnBlock(this.browser, 'simple_circle_1');
7168
await this.browser.pause(PAUSE_TIME);
7269
await keyUp(this.browser);
@@ -77,8 +74,6 @@ suite('Keyboard navigation on Blocks', function () {
7774
});
7875

7976
test('Down from parent block selects first child block', async function () {
80-
await tabNavigateToWorkspace(this.browser);
81-
await this.browser.pause(PAUSE_TIME);
8277
await focusOnBlock(this.browser, 'p5_setup_1');
8378
await this.browser.pause(PAUSE_TIME);
8479
await keyDown(this.browser);
@@ -88,8 +83,6 @@ suite('Keyboard navigation on Blocks', function () {
8883
});
8984

9085
test('Up from child block selects parent block', async function () {
91-
await tabNavigateToWorkspace(this.browser);
92-
await this.browser.pause(PAUSE_TIME);
9386
await focusOnBlock(this.browser, 'p5_canvas_1');
9487
await this.browser.pause(PAUSE_TIME);
9588
await keyUp(this.browser);
@@ -99,8 +92,6 @@ suite('Keyboard navigation on Blocks', function () {
9992
});
10093

10194
test('Right from block selects first field', async function () {
102-
await tabNavigateToWorkspace(this.browser);
103-
await this.browser.pause(PAUSE_TIME);
10495
await focusOnBlock(this.browser, 'p5_canvas_1');
10596
await this.browser.pause(PAUSE_TIME);
10697
await keyRight(this.browser);
@@ -113,8 +104,6 @@ suite('Keyboard navigation on Blocks', function () {
113104
});
114105

115106
test('Right from block selects first inline input', async function () {
116-
await tabNavigateToWorkspace(this.browser);
117-
await this.browser.pause(PAUSE_TIME);
118107
await focusOnBlock(this.browser, 'simple_circle_1');
119108
await this.browser.pause(PAUSE_TIME);
120109
await keyRight(this.browser);
@@ -126,8 +115,6 @@ suite('Keyboard navigation on Blocks', function () {
126115
});
127116

128117
test('Up from inline input selects statement block', async function () {
129-
await tabNavigateToWorkspace(this.browser);
130-
await this.browser.pause(PAUSE_TIME);
131118
await focusOnBlock(this.browser, 'math_number_2');
132119
await this.browser.pause(PAUSE_TIME);
133120
await keyUp(this.browser);
@@ -139,8 +126,6 @@ suite('Keyboard navigation on Blocks', function () {
139126
});
140127

141128
test('Left from first inline input selects block', async function () {
142-
await tabNavigateToWorkspace(this.browser);
143-
await this.browser.pause(PAUSE_TIME);
144129
await focusOnBlock(this.browser, 'math_number_2');
145130
await this.browser.pause(PAUSE_TIME);
146131
await keyLeft(this.browser);
@@ -152,8 +137,6 @@ suite('Keyboard navigation on Blocks', function () {
152137
});
153138

154139
test('Right from first inline input selects second inline input', async function () {
155-
await tabNavigateToWorkspace(this.browser);
156-
await this.browser.pause(PAUSE_TIME);
157140
await focusOnBlock(this.browser, 'math_number_2');
158141
await this.browser.pause(PAUSE_TIME);
159142
await keyRight(this.browser);
@@ -165,8 +148,6 @@ suite('Keyboard navigation on Blocks', function () {
165148
});
166149

167150
test('Left from second inline input selects first inline input', async function () {
168-
await tabNavigateToWorkspace(this.browser);
169-
await this.browser.pause(PAUSE_TIME);
170151
await focusOnBlock(this.browser, 'math_number_3');
171152
await this.browser.pause(PAUSE_TIME);
172153
await keyLeft(this.browser);
@@ -178,8 +159,6 @@ suite('Keyboard navigation on Blocks', function () {
178159
});
179160

180161
test('Right from last inline input selects next block', async function () {
181-
await tabNavigateToWorkspace(this.browser);
182-
await this.browser.pause(PAUSE_TIME);
183162
await focusOnBlock(this.browser, 'colour_picker_1');
184163
await this.browser.pause(PAUSE_TIME);
185164
await keyRight(this.browser);
@@ -190,8 +169,6 @@ suite('Keyboard navigation on Blocks', function () {
190169
});
191170

192171
test('Down from inline input selects next block', async function () {
193-
await tabNavigateToWorkspace(this.browser);
194-
await this.browser.pause(PAUSE_TIME);
195172
await focusOnBlock(this.browser, 'colour_picker_1');
196173
await this.browser.pause(PAUSE_TIME);
197174
await keyDown(this.browser);
@@ -202,8 +179,6 @@ suite('Keyboard navigation on Blocks', function () {
202179
});
203180

204181
test("Down from inline input selects block's child block", async function () {
205-
await tabNavigateToWorkspace(this.browser);
206-
await this.browser.pause(PAUSE_TIME);
207182
await focusOnBlock(this.browser, 'logic_boolean_1');
208183
await this.browser.pause(PAUSE_TIME);
209184
await keyDown(this.browser);
@@ -214,8 +189,6 @@ suite('Keyboard navigation on Blocks', function () {
214189
});
215190

216191
test('Right from text block selects shadow block then field', async function () {
217-
await tabNavigateToWorkspace(this.browser);
218-
await this.browser.pause(PAUSE_TIME);
219192
await focusOnBlock(this.browser, 'text_print_1');
220193
await this.browser.pause(PAUSE_TIME);
221194
await keyRight(this.browser);
@@ -236,33 +209,27 @@ suite('Keyboard navigation on Blocks', function () {
236209
});
237210

238211
test('Losing focus cancels move', async function () {
239-
await tabNavigateToWorkspace(this.browser);
240-
await this.browser.pause(PAUSE_TIME);
241212
await focusOnBlock(this.browser, 'text_print_1');
242-
await this.browser.keys('m');
243-
await this.browser.pause(PAUSE_TIME);
213+
await sendKeyAndWait(this.browser, 'm');
244214

245215
chai.assert.isTrue(await isDragging(this.browser));
246216

247-
await this.browser.keys(Key.Tab);
248-
await this.browser.pause(PAUSE_TIME);
217+
await sendKeyAndWait(this.browser, Key.Tab);
249218

250219
chai.assert.isFalse(await isDragging(this.browser));
251220
});
252221
});
253222

254223
suite('Keyboard navigation on Fields', function () {
255-
// Setting timeout to unlimited as these tests take a longer time to run than most mocha test
256-
this.timeout(0);
224+
// Disable timeouts when non-zero PAUSE_TIME is used to watch tests run.
225+
if (PAUSE_TIME) this.timeout(0);
257226

258-
// Setup Selenium for all of the tests
227+
// Clear the workspace and load start blocks.
259228
suiteSetup(async function () {
260229
this.browser = await testSetup(testFileLocations.NAVIGATION_TEST_BLOCKS);
261230
});
262231

263232
test('Up from first field selects block', async function () {
264-
await tabNavigateToWorkspace(this.browser);
265-
await this.browser.pause(PAUSE_TIME);
266233
await focusOnBlockField(this.browser, 'p5_canvas_1', 'WIDTH');
267234
await this.browser.pause(PAUSE_TIME);
268235
await keyUp(this.browser);
@@ -274,8 +241,6 @@ suite('Keyboard navigation on Fields', function () {
274241
});
275242

276243
test('Left from first field selects block', async function () {
277-
await tabNavigateToWorkspace(this.browser);
278-
await this.browser.pause(PAUSE_TIME);
279244
await focusOnBlockField(this.browser, 'p5_canvas_1', 'WIDTH');
280245
await this.browser.pause(PAUSE_TIME);
281246
await keyLeft(this.browser);
@@ -287,8 +252,6 @@ suite('Keyboard navigation on Fields', function () {
287252
});
288253

289254
test('Right from first field selects second field', async function () {
290-
await tabNavigateToWorkspace(this.browser);
291-
await this.browser.pause(PAUSE_TIME);
292255
await focusOnBlockField(this.browser, 'p5_canvas_1', 'WIDTH');
293256
await this.browser.pause(PAUSE_TIME);
294257
await keyRight(this.browser);
@@ -301,8 +264,6 @@ suite('Keyboard navigation on Fields', function () {
301264
});
302265

303266
test('Left from second field selects first field', async function () {
304-
await tabNavigateToWorkspace(this.browser);
305-
await this.browser.pause(PAUSE_TIME);
306267
await focusOnBlockField(this.browser, 'p5_canvas_1', 'HEIGHT');
307268
await this.browser.pause(PAUSE_TIME);
308269
await keyLeft(this.browser);
@@ -315,8 +276,6 @@ suite('Keyboard navigation on Fields', function () {
315276
});
316277

317278
test('Right from second field selects next block', async function () {
318-
await tabNavigateToWorkspace(this.browser);
319-
await this.browser.pause(PAUSE_TIME);
320279
await focusOnBlockField(this.browser, 'p5_canvas_1', 'HEIGHT');
321280
await this.browser.pause(PAUSE_TIME);
322281
await keyRight(this.browser);
@@ -327,8 +286,6 @@ suite('Keyboard navigation on Fields', function () {
327286
});
328287

329288
test('Down from field selects next block', async function () {
330-
await tabNavigateToWorkspace(this.browser);
331-
await this.browser.pause(PAUSE_TIME);
332289
await focusOnBlockField(this.browser, 'p5_canvas_1', 'WIDTH');
333290
await this.browser.pause(PAUSE_TIME);
334291
await keyDown(this.browser);
@@ -339,8 +296,6 @@ suite('Keyboard navigation on Fields', function () {
339296
});
340297

341298
test("Down from field selects block's child block", async function () {
342-
await tabNavigateToWorkspace(this.browser);
343-
await this.browser.pause(PAUSE_TIME);
344299
await focusOnBlockField(this.browser, 'controls_repeat_1', 'TIMES');
345300
await this.browser.pause(PAUSE_TIME);
346301
await keyDown(this.browser);
@@ -354,8 +309,7 @@ suite('Keyboard navigation on Fields', function () {
354309
// Open a field editor dropdown
355310
await focusOnBlockField(this.browser, 'logic_boolean_1', 'BOOL');
356311
await this.browser.pause(PAUSE_TIME);
357-
await this.browser.keys(Key.Enter);
358-
await this.browser.pause(PAUSE_TIME);
312+
await sendKeyAndWait(this.browser, Key.Enter);
359313

360314
// Try to navigate to a different block
361315
await keyRight(this.browser);
@@ -368,13 +322,12 @@ suite('Keyboard navigation on Fields', function () {
368322
// Open colour picker
369323
await focusOnBlockField(this.browser, 'colour_picker_1', 'COLOUR');
370324
await this.browser.pause(PAUSE_TIME);
371-
await this.browser.keys(Key.Enter);
372-
await this.browser.pause(PAUSE_TIME);
325+
await sendKeyAndWait(this.browser, Key.Enter);
373326

374327
// Move right to pick a new colour.
375328
await keyRight(this.browser);
376329
// Enter to choose.
377-
await this.browser.keys(Key.Enter);
330+
await sendKeyAndWait(this.browser, Key.Enter);
378331

379332
// Focus seems to take longer than a single pause to settle.
380333
await this.browser.waitUntil(

‎test/webdriverio/test/block_comment_test.ts‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,14 +13,15 @@ import {
1313
sendKeyAndWait,
1414
testFileLocations,
1515
keyRight,
16+
PAUSE_TIME,
1617
} from './test_setup.js';
1718
import {Key} from 'webdriverio';
1819

1920
suite('Block comment navigation', function () {
20-
// Setting timeout to unlimited as these tests take a longer time to run than most mocha test
21-
this.timeout(0);
21+
// Disable timeouts when non-zero PAUSE_TIME is used to watch tests run.
22+
if (PAUSE_TIME) this.timeout(0);
2223

23-
// Setup Selenium for all of the tests
24+
// Clear the workspace and load start blocks.
2425
setup(async function () {
2526
this.browser = await testSetup(testFileLocations.NAVIGATION_TEST_BLOCKS);
2627
await this.browser.execute(() => {

0 commit comments

Comments
 (0)