Skip to content

test: Make tests faster, more robust, more consistent #660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cpcallen wants to merge 8 commits into from
Closed

test: Make tests faster, more robust, more consistent #660

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
12ac3cd
fix(tests): Restore window size after scroll tests
cpcallen Jul 10, 2025
6bd74be
docs(tests): "Selenium" -> "WebdriverIO"
cpcallen Jul 10, 2025
0e57632
chore(tests): Remove unneeded timeout(0) calls
cpcallen Jul 10, 2025
0feffcc
fix(tests): Set PAUSE_TIME to 0
cpcallen Jul 10, 2025
074c50a
fix(tests): Fix flakey right-click test
cpcallen Jul 10, 2025
7b9a026
refactor(tests): Use sendKeyAndWait where appropriate
cpcallen Jul 10, 2025
7c73fc3
chore: Format
cpcallen Jul 10, 2025
80229d6
chore: Lint
cpcallen Jul 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 8 additions & 13 deletions test/webdriverio/test/actions_test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,12 @@ import {
tabNavigateToWorkspace,
testFileLocations,
testSetup,
sendKeyAndWait,
keyRight,
contextMenuItems,
} from './test_setup.js';

suite('Menus test', function () {
// Setting timeout to unlimited as these tests take longer time to run
this.timeout(0);

// Clear the workspace and load start blocks
setup(async function () {
this.browser = await testSetup(testFileLocations.BASE);
Expand All @@ -33,8 +31,7 @@ suite('Menus test', function () {
await tabNavigateToWorkspace(this.browser);
await focusOnBlock(this.browser, 'draw_circle_1');
await this.browser.pause(PAUSE_TIME);
await this.browser.keys([Key.Ctrl, Key.Return]);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, [Key.Ctrl, Key.Return]);

chai.assert.deepEqual(
process.platform === 'darwin'
Expand Down Expand Up @@ -76,8 +73,7 @@ suite('Menus test', function () {
await moveToToolboxCategory(this.browser, 'Functions');
// Move to flyout.
await keyRight(this.browser);
await this.browser.keys([Key.Ctrl, Key.Return]);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, [Key.Ctrl, Key.Return]);

chai.assert.deepEqual(
process.platform === 'darwin'
Expand All @@ -100,9 +96,8 @@ suite('Menus test', function () {
test('Menu on workspace', async function () {
// Navigate to draw_circle_1.
await tabNavigateToWorkspace(this.browser);
await this.browser.keys('w');
await this.browser.keys([Key.Ctrl, Key.Return]);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, 'w');
await sendKeyAndWait(this.browser, [Key.Ctrl, Key.Return]);

chai.assert.deepEqual(
process.platform === 'darwin'
Expand Down Expand Up @@ -135,9 +130,9 @@ suite('Menus test', function () {
await tabNavigateToWorkspace(this.browser);
await focusOnBlock(this.browser, 'draw_circle_1');
// Start moving the block
await this.browser.keys('m');
await this.browser.keys([Key.Ctrl, Key.Return]);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, 'm');
await sendKeyAndWait(this.browser, [Key.Ctrl, Key.Return]);

chai.assert.isTrue(
await contextMenuExists(this.browser, 'Collapse Block', true),
'The menu should not be openable during a move',
Expand Down
25 changes: 8 additions & 17 deletions test/webdriverio/test/basic_test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import {
testSetup,
testFileLocations,
PAUSE_TIME,
sendKeyAndWait,
tabNavigateToWorkspace,
keyLeft,
keyRight,
Expand All @@ -25,10 +26,7 @@ import {
import {Key} from 'webdriverio';

suite('Keyboard navigation on Blocks', function () {
// Setting timeout to unlimited as these tests take a longer time to run than most mocha test
this.timeout(0);

// Setup Selenium for all of the tests
// Clear the workspace and load start blocks.
suiteSetup(async function () {
this.browser = await testSetup(testFileLocations.NAVIGATION_TEST_BLOCKS);
});
Expand Down Expand Up @@ -239,23 +237,18 @@ suite('Keyboard navigation on Blocks', function () {
await tabNavigateToWorkspace(this.browser);
await this.browser.pause(PAUSE_TIME);
await focusOnBlock(this.browser, 'text_print_1');
await this.browser.keys('m');
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, 'm');

chai.assert.isTrue(await isDragging(this.browser));

await this.browser.keys(Key.Tab);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Tab);

chai.assert.isFalse(await isDragging(this.browser));
});
});

suite('Keyboard navigation on Fields', function () {
// Setting timeout to unlimited as these tests take a longer time to run than most mocha test
this.timeout(0);

// Setup Selenium for all of the tests
// Clear the workspace and load start blocks.
Copy link
Collaborator

@maribethb maribethb Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that unlike the rest of the test files, this one only does this before the entire suite, not before each test. Possibly worth either changing that or adding a comment.

Copy link
Collaborator Author

@cpcallen cpcallen Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point re: documentation. I had noticed the setup vs. suiteSetup inconsistency. While looking in to that I realised that the tests in this suite are almost certainly passing only because of a combination of #632 and the fact that there happen to be a total of exactly the right number of tab stops in the document that tabNavigateToWorkspace loops focus back to where it started.

suiteSetup(async function () {
this.browser = await testSetup(testFileLocations.NAVIGATION_TEST_BLOCKS);
});
Expand Down Expand Up @@ -354,8 +347,7 @@ suite('Keyboard navigation on Fields', function () {
// Open a field editor dropdown
await focusOnBlockField(this.browser, 'logic_boolean_1', 'BOOL');
await this.browser.pause(PAUSE_TIME);
await this.browser.keys(Key.Enter);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Enter);

// Try to navigate to a different block
await keyRight(this.browser);
Expand All @@ -368,13 +360,12 @@ suite('Keyboard navigation on Fields', function () {
// Open colour picker
await focusOnBlockField(this.browser, 'colour_picker_1', 'COLOUR');
await this.browser.pause(PAUSE_TIME);
await this.browser.keys(Key.Enter);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Enter);

// Move right to pick a new colour.
await keyRight(this.browser);
// Enter to choose.
await this.browser.keys(Key.Enter);
await sendKeyAndWait(this.browser, Key.Enter);

// Focus seems to take longer than a single pause to settle.
await this.browser.waitUntil(
Expand Down
18 changes: 7 additions & 11 deletions test/webdriverio/test/clipboard_test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,11 @@ import {
focusOnBlock,
focusOnBlockField,
blockIsPresent,
sendKeyAndWait,
} from './test_setup.js';
import {Key, KeyAction, PointerAction, WheelAction} from 'webdriverio';

suite('Clipboard test', function () {
// Setting timeout to unlimited as these tests take longer time to run
this.timeout(0);

// Clear the workspace and load start blocks
setup(async function () {
this.browser = await testSetup(testFileLocations.BASE);
Expand All @@ -36,9 +34,8 @@ suite('Clipboard test', function () {
await focusOnBlock(this.browser, 'draw_circle_1');

// Copy and paste
await this.browser.keys([Key.Ctrl, 'c']);
await this.browser.keys([Key.Ctrl, 'v']);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, [Key.Ctrl, 'c']);
await sendKeyAndWait(this.browser, [Key.Ctrl, 'v']);

const block = await getBlockElementById(this.browser, 'draw_circle_1');
const blocks = await getSameBlocks(this.browser, block);
Expand All @@ -58,9 +55,9 @@ suite('Clipboard test', function () {
const block = await getBlockElementById(this.browser, 'draw_circle_1');

// Cut and paste
await this.browser.keys([Key.Ctrl, 'x']);
await sendKeyAndWait(this.browser, [Key.Ctrl, 'x']);
await block.waitForExist({reverse: true});
await this.browser.keys([Key.Ctrl, 'v']);
await sendKeyAndWait(this.browser, [Key.Ctrl, 'v']);
await block.waitForExist();
await this.browser.pause(PAUSE_TIME);

Expand Down Expand Up @@ -118,11 +115,10 @@ suite('Clipboard test', function () {
// Open a field editor
await focusOnBlockField(this.browser, 'draw_circle_1_color', 'COLOUR');
await this.browser.pause(PAUSE_TIME);
await this.browser.keys(Key.Enter);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Enter);

// Try to cut block while field editor is open
await this.browser.keys([Key.Ctrl, 'x']);
await sendKeyAndWait(this.browser, [Key.Ctrl, 'x']);

// Block is not deleted
chai.assert.isTrue(
Expand Down
45 changes: 15 additions & 30 deletions test/webdriverio/test/delete_test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,13 @@ import {
testFileLocations,
PAUSE_TIME,
tabNavigateToWorkspace,
sendKeyAndWait,
keyRight,
focusOnBlockField,
} from './test_setup.js';
import {Key} from 'webdriverio';

suite('Deleting Blocks', function () {
// Setting timeout to unlimited as these tests take a longer time to run than most mocha test
this.timeout(0);

setup(async function () {
this.browser = await testSetup(testFileLocations.NAVIGATION_TEST_BLOCKS);
await this.browser.pause(PAUSE_TIME);
Expand All @@ -39,8 +37,7 @@ suite('Deleting Blocks', function () {
.expect(await blockIsPresent(this.browser, 'controls_if_2'))
.equal(true);

await this.browser.keys(Key.Backspace);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Backspace);

chai
.expect(await blockIsPresent(this.browser, 'controls_if_2'))
Expand All @@ -61,8 +58,7 @@ suite('Deleting Blocks', function () {
.expect(await blockIsPresent(this.browser, 'controls_if_2'))
.equal(true);

await this.browser.keys([Key.Ctrl, 'x']);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, [Key.Ctrl, 'x']);

chai
.expect(await blockIsPresent(this.browser, 'controls_if_2'))
Expand All @@ -84,8 +80,7 @@ suite('Deleting Blocks', function () {
.equal(true);
chai.expect(await blockIsPresent(this.browser, 'text_print_1')).equal(true);

await this.browser.keys(Key.Backspace);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Backspace);

chai
.expect(await blockIsPresent(this.browser, 'logic_boolean_1'))
Expand All @@ -106,8 +101,7 @@ suite('Deleting Blocks', function () {
.equal(true);
chai.expect(await blockIsPresent(this.browser, 'text_print_1')).equal(true);

await this.browser.keys([Key.Ctrl, 'x']);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, [Key.Ctrl, 'x']);

chai
.expect(await blockIsPresent(this.browser, 'logic_boolean_1'))
Expand All @@ -127,8 +121,7 @@ suite('Deleting Blocks', function () {
.expect(await blockIsPresent(this.browser, 'logic_boolean_1'))
.equal(true);

await this.browser.keys(Key.Backspace);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Backspace);

chai
.expect(await blockIsPresent(this.browser, 'logic_boolean_1'))
Expand All @@ -149,8 +142,7 @@ suite('Deleting Blocks', function () {
.expect(await blockIsPresent(this.browser, 'logic_boolean_1'))
.equal(true);

await this.browser.keys([Key.Ctrl, 'x']);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, [Key.Ctrl, 'x']);

chai
.expect(await blockIsPresent(this.browser, 'logic_boolean_1'))
Expand All @@ -176,16 +168,13 @@ suite('Deleting Blocks', function () {
// Move to flyout.
await keyRight(this.browser);
// Select number block.
await this.browser.keys(Key.Enter);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Enter);
// Confirm move.
await this.browser.keys(Key.Enter);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Enter);

chai.assert.equal('math_number', await getFocusedBlockType(this.browser));

await this.browser.keys(Key.Backspace);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Backspace);

chai.assert.equal(
await getCurrentFocusedBlockId(this.browser),
Expand All @@ -203,16 +192,13 @@ suite('Deleting Blocks', function () {
// Move to flyout.
await keyRight(this.browser);
// Select number block.
await this.browser.keys(Key.Enter);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Enter);
// Confirm move.
await this.browser.keys(Key.Enter);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Enter);

chai.assert.equal('math_number', await getFocusedBlockType(this.browser));

await this.browser.keys([Key.Ctrl, 'x']);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, [Key.Ctrl, 'x']);

chai.assert.equal(
await getCurrentFocusedBlockId(this.browser),
Expand All @@ -224,11 +210,10 @@ suite('Deleting Blocks', function () {
// Open a field editor
await focusOnBlockField(this.browser, 'colour_picker_1', 'COLOUR');
await this.browser.pause(PAUSE_TIME);
await this.browser.keys(Key.Enter);
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, Key.Enter);

// Try to delete block while field editor is open
await this.browser.keys(Key.Backspace);
await sendKeyAndWait(this.browser, Key.Backspace);

// Block is not deleted
chai.assert.isTrue(await blockIsPresent(this.browser, 'colour_picker_1'));
Expand Down
9 changes: 3 additions & 6 deletions test/webdriverio/test/duplicate_test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,10 @@ import {
tabNavigateToWorkspace,
testFileLocations,
testSetup,
sendKeyAndWait,
} from './test_setup.js';

suite('Duplicate test', function () {
// Setting timeout to unlimited as these tests take longer time to run
this.timeout(0);

// Clear the workspace and load start blocks
setup(async function () {
this.browser = await testSetup(testFileLocations.BASE);
Expand All @@ -32,8 +30,7 @@ suite('Duplicate test', function () {
await focusOnBlock(this.browser, 'draw_circle_1');

// Duplicate
await this.browser.keys('d');
await this.browser.pause(PAUSE_TIME);
await sendKeyAndWait(this.browser, 'd');

// Check a different block of the same type has focus.
chai.assert.notEqual(
Expand Down Expand Up @@ -65,7 +62,7 @@ suite('Duplicate test', function () {
await this.browser.pause(PAUSE_TIME);

// Duplicate.
await this.browser.keys('d');
await sendKeyAndWait(this.browser, 'd');

// Assert we have two comments with the same text.
const commentTexts = await this.browser.execute(() =>
Expand Down
3 changes: 2 additions & 1 deletion test/webdriverio/test/flyout_test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import {
keyDown,
tabNavigateBackward,
tabNavigateToWorkspace,
sendKeyAndWait,
keyRight,
getCurrentFocusNodeId,
getCurrentFocusedBlockId,
Expand Down Expand Up @@ -164,7 +165,7 @@ suite('Toolbox and flyout test', function () {
test('Tabbing to the workspace after selecting flyout block via workspace toolbox shortcut should close the flyout', async function () {
await tabNavigateToWorkspace(this.browser);

await this.browser.keys('t');
await sendKeyAndWait(this.browser, 't');
await keyRight(this.browser);
await tabNavigateForward(this.browser);

Expand Down
Loading
Loading