runTests in programmatic API does not throw errors

[alechirsch]

Describe the bug

When using the programmatic API, the runTests function does not throw an error if the test fails, nor does it return any data about the tests that ran.

Sample test

You can see my docker setup as described here #2483

I did not expect runTests to return anything since that is not documented, however that might be a good feature to add

const Nightwatch = require('nightwatch');


(async function() {
	const runner = Nightwatch.CliRunner({
		config: './nightwatch.conf.js',
		env: 'default'
	});

	// here you can overwrite any nightwatch config setting
	const settings = {
		src_folders: ['test']
	};
	runner.setup(settings);

	await runner.startWebDriver();

	try {
		let testResults = await runner.runTests();
		console.log(testResults);
	}
	catch (err) {
		console.log('I ERRRORRED');
		console.error('An error occurred:', err);
	}

	await runner.stopWebDriver();
}());

Successful test

module.exports = {
	'Demo test Google' (browser) {
		browser
			.url('http://www.google.com')
			.waitForElementVisible('body', 2000);
		browser.end();
	}
};

Unsuccessful test

module.exports = {
	'Demo test Google' (browser) {
		browser
			.url('http://www.google.com')
			.waitForElementVisible('bodya', 2000);
		browser.end();
	}
};

The result is the same for both tests, runTests does not throw an error for the unsuccessful test. I can see the error in the console from nightwatch, however the catch does not get hit and the text 'I ERRRORRED' is not printed.

Verbose output

node app.js 

[Test Script] Test Suite
========================
ℹ Connected to localhost on port 9515 (1149ms).
  Using: chrome (85.0.4183.121) on Linux platform.

Running:  Demo test Google

✖ Timed out while waiting for element <bodya> to be present for 2000 milliseconds. - expected "visible" but got: "not found" (2105ms)
    at Object.Demo test Google (/src/test/test_script.js:7:5)
    at processTicksAndRejections (internal/process/task_queues.js:93:5) 


FAILED: 1 assertions failed (3.983s)
_________________________________________________

TEST FAILURE:  1 assertions failed, 0 passed (5.567s)

 ✖ test_script
 – Demo test Google (3.983s)
   Timed out while waiting for element <bodya> to be present for 2000 milliseconds. - expected "visible" but got: "not found" (2105ms)
       at Object.Demo test Google (/src/test/test_script.js:7:5)
       at processTicksAndRejections (internal/process/task_queues.js:93:5)

Configuration

module.exports = {
	webdriver: {
		start_process: true,
		server_path: '/usr/bin/chromedriver',
		port: 9515
	},
	test_settings: {
		default: {
			desiredCapabilities: {
				browserName: 'chrome',
				chromeOptions: {
					binary: '/usr/bin/chromium-browser',
					args: ['--no-sandbox']
				}
			}
		}
	}
};

Your Environment

You can see my docker setup as described here #2483

[alechirsch]

Digging through code sample from nightwatches unit test I found that I could do this:

/* eslint-disable no-console */
const Nightwatch = require('nightwatch');


(async function() {
	const runner = Nightwatch.CliRunner({
		config: './nightwatch.conf.js',
		env: 'default'
	});

	// here you can overwrite any nightwatch config setting
	let nightwatchResult = null;
	const settings = {
		src_folders: ['test'],
		globals: {
			reporter(results, cb) {
				nightwatchResult = results;
				cb();
			}
		}
	};
	runner.setup(settings);

	await runner.startWebDriver();

	try {
		let testResults = await runner.runTests();
		console.log(nightwatchResult)
	}
	catch (err) {
		console.log('I ERRRORRED');
		console.error('An error occurred:', err);
	}

	await runner.stopWebDriver();
}());

Added a custom reporter to set a variable in the parent context. I would still like to propose that by default, runnner.runTests returns the results of the tests.

[stale]

This issue has been automatically marked as stale because it has not had any recent activity.
If possible, please retry using the latest Nightwatch version and update the issue with any relevant details. If no further activity occurs, it will be closed. Thank you for your contribution.

[alechirsch]

Bump, not stale

[stale]

This issue has been automatically marked as stale because it has not had any recent activity.
If possible, please retry using the latest Nightwatch version and update the issue with any relevant details. If no further activity occurs, it will be closed. Thank you for your contribution.

[tim-yao]

Hi @swrdfish @beatfactor This is still an issue in Nightwatch v2.5.1.
Here is an example for v2.
Using https://github.com/nightwatchjs/nightwatch-examples, create a file: test.js with the below code.
Then run node test.js. You will see testAPI and testEnsure won't be able to catch any error. But testExpect and testAssert work correctly.

const Nightwatch = require('nightwatch');

const client = Nightwatch.createClient({
  headless: true,
  output: true,
  silent: true, // set to false to enable verbose logging
  browserName: 'chrome', // can be either: firefox, chrome, safari, or edge

  // set the global timeout to be used with waitFor commands and when retrying assertions/expects
  timeout: 10000,

  // set the current test environment from the nightwatch config
  env: null,

  // any additional capabilities needed
  desiredCapabilities: {

  },

  // can define/overwrite test globals here; 
  // when using a third-party test runner only the global hooks onBrowserNavigate/onBrowserQuit are supported
  globals: {},

  // when the test runner used supports running tests in parallel; 
  // set to true if you need the webdriver port to be randomly generated
  parallel: false, 

  // All other Nightwatch config settings can be overwritten here, such as:
  disable_colors: false
});



async function testAPI(browser) {
  console.log('========= Test API =========')
  try {
    await browser.url('https://www.google.com')
    await browser.waitForElementVisible('#no-such-thing', 1000)
    await browser.click('#no-such-thing')
    console.info('❓No error be thrown, it should throw error')
  } catch (err) {
    console.error('👍 Yes, we get the expected error:', err)
  }
}

async function testEnsure(browser) {
  console.log('========= Test Ensure =========')
  try {
    await browser.url('https://www.google.com')
    await browser.ensure.elementIsVisible('#not-existing-element')
    console.info('❓No error be thrown, it should throw error')
  } catch (err) {
    console.error('👍 Yes, we get the expected error:', err)
  }
}

async function testAssert(browser) {
  console.log('========= Test Assert =========')
  try {
    await browser.url('https://www.google.com')
    await browser.assert.visible('#not-existing-element')
    console.info('❓No error be thrown, it should throw error')
  } catch (err) {
    console.error('👍 Yes, we get the expected error:', err)
  }
}

async function testExpect(browser) {
  console.log('========= Test Expect =========')
  try {
    await browser.url('https://www.google.com')
    await browser.expect.element('#not-existing-element').to.be.present
    console.info('❓No error be thrown, it should throw error')
  } catch (err) {
    console.error('👍 Yes, we get the expected error:', err)
  }
}

async function testAll() {
  const browser = await client.launchBrowser()
  await testAPI(browser)
  await testEnsure(browser)
  await testAssert(browser)
  await testExpect(browser)
  browser.quit()
}

testAll()

This issue causes any test runner using Nightwatch v2 programmatic API won't able to fail the test steps like in my Cucumber-js integration nightwatchjs-community/cucumber-nightwatch#9

It's a critical issue for me, so please help if you have time. Thanks.