You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CHANGELOG.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -71,7 +71,7 @@ Scenario(..., ({ I }) => {
71
71
72
72
Previously `tryTo` and `retryTo` were available globally via plugins. This behavior is deprecated as of 3.7 and will be removed in 4.0. Import these functions via effects instead. Similarly, `within` will be moved to `effects` in 4.0.
This AI plugin analyzes failures in test runs and provides brief summaries. For more than 5 failures, it performs cluster analysis and aggregates failures into groups, attempting to find common causes. It is recommended to use Deepseek R1 model or OpenAI o3 for better reasoning on clustering:
104
+
This [AI plugin](./plugins#analyze) analyzes failures in test runs and provides brief summaries. For more than 5 failures, it performs cluster analysis and aggregates failures into groups, attempting to find common causes. It is recommended to use Deepseek R1 model or OpenAI o3 for better reasoning on clustering:
105
105
106
106
```js
107
107
• SUMMARY The test failed because the expected text "Sign in" was not found on the page, indicating a possible issue with HTML elements or their visibility.
Copy file name to clipboardExpand all lines: docs/plugins.md
+63-26Lines changed: 63 additions & 26 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,13 +5,50 @@ sidebar: auto
5
5
title: Plugins
6
6
---
7
7
8
-
<!-- Generated by documentation.js. Update this documentation by updating the source code. -->## analyze### Parameters* `config` **any** (optional, default `{}`)## authLogs user in for the first test and reuses session for next tests.
8
+
<!-- Generated by documentation.js. Update this documentation by updating the source code. -->## analyzeCodeceptJS Analyze Plugin - Uses AI to analyze test failures and provide insightsThis plugin analyzes failed tests using AI to provide detailed explanations and group similar failures.
9
9
10
-
Works by saving cookies into memory or file.
11
-
If a session expires automatically logs in again.> For better development experience cookies can be saved into file, so a session can be reused while writing tests.#### Usage1. Enable this plugin and configure as described below 2. Define user session names (example: `user`, `editor`, `admin`, etc). 3. Define how users are logged in and how to check that user is logged in 4. Use `login` object inside your tests to log in:```js
12
-
// inside a test file
13
-
// use login to inject auto-login function
14
-
Feature('Login');
10
+
When enabled with --ai flag, it generates reports after test execution.#### Usage`js
11
+
// in codecept.conf.js
12
+
exports.config = {
13
+
plugins: {
14
+
analyze: {
15
+
enabled: true,
16
+
clusterize: 5,
17
+
analyze: 2,
18
+
vision: false
19
+
}
20
+
}
21
+
}
22
+
`#### Configuration\* `clusterize` (number) - minimum number of failures to trigger clustering analysis. Default: 5
23
+
24
+
-`analyze` (number) - maximum number of individual test failures to analyze in detail. Default: 2
25
+
-`vision` (boolean) - enables visual analysis of test screenshots. Default: false
26
+
-`categories` (array) - list of failure categories for classification. Defaults to:
27
+
- Browser connection error / browser crash
28
+
- Network errors (server error, timeout, etc)
29
+
- HTML / page elements (not found, not visible, etc)
30
+
- Navigation errors (404, etc)
31
+
- Code errors (syntax error, JS errors, etc)
32
+
- Library & framework errors
33
+
- Data errors (password incorrect, invalid format, etc)
34
+
- Assertion failures
35
+
- Other errors
36
+
-`prompts` (object) - customize AI prompts for analysis
37
+
-`clusterize` - prompt for clustering analysis
38
+
-`analyze` - prompt for individual test analysis#### Features\* Groups similar failures when number of failures >= clusterize value
39
+
- Provides detailed analysis of individual failures
40
+
- Analyzes screenshots if vision=true and screenshots are available
41
+
- Classifies failures into predefined categories
42
+
- Suggests possible causes and solutions### Parameters\*`config`**[Object][1]** Plugin configuration (optional, default `{}`)Returns **void** ## authLogs user in for the first test and reuses session for next tests.
43
+
Works by saving cookies into memory or file.
44
+
If a session expires automatically logs in again.> For better development experience cookies can be saved into file, so a session can be reused while writing tests.#### Usage1. Enable this plugin and configure as described below
45
+
46
+
2. Define user session names (example: `user`, `editor`, `admin`, etc).
47
+
3. Define how users are logged in and how to check that user is logged in
48
+
4. Use `login` object inside your tests to log in:```js
49
+
// inside a test file
50
+
// use login to inject auto-login function
51
+
Feature('Login');
15
52
16
53
Before(({ login }) => {
17
54
login('user'); // login using user session
@@ -298,11 +335,11 @@ plugins: {
298
335
outputDir: 'output/coverage'
299
336
}
300
337
}
301
-
```Possible config options, More could be found at [monocart-coverage-reports][1]* `debug`: debug info. By default, false.
338
+
```Possible config options, More could be found at [monocart-coverage-reports][2]* `debug`: debug info. By default, false.
302
339
* `name`: coverage report name.
303
340
* `outputDir`: path to coverage report.
304
341
* `sourceFilter`: filter the source files.
305
-
* `sourcePath`: option to resolve a custom path.### Parameters* `config`  ## customLocatorCreates a [custom locator][2] by using special attributes in HTML.If you have a convention to use `data-test-id` or `data-qa` attributes to mark active elements for e2e tests,
342
+
* `sourcePath`: option to resolve a custom path.### Parameters* `config`  ## customLocatorCreates a [custom locator][3] by using special attributes in HTML.If you have a convention to use `data-test-id` or `data-qa` attributes to mark active elements for e2e tests,
306
343
you can enable this plugin to simplify matching elements with these attributes:```js
307
344
// replace this:
308
345
I.click({ css: '[data-test-id=register_button]');
@@ -380,13 +417,13 @@ await eachElement('check all items are visible', '.item', async (el) => {
380
417
assert(await el.isVisible());
381
418
});
382
419
```This method works with WebDriver, Playwright, Puppeteer, Appium helpers.Function parameter `el` represents a matched element.
383
-
Depending on a helper API of `el` can be different. Refer to API of corresponding browser testing engine for a complete API list:* [Playwright ElementHandle][3]
384
-
* [Puppeteer][4]
385
-
* [webdriverio element][5]#### Configuration* `registerGlobal` - to register `eachElement` function globally, true by defaultIf `registerGlobal` is false you can use eachElement from the plugin:```js
420
+
Depending on a helper API of `el` can be different. Refer to API of corresponding browser testing engine for a complete API list:* [Playwright ElementHandle][4]
421
+
* [Puppeteer][5]
422
+
* [webdriverio element][6]#### Configuration* `registerGlobal` - to register `eachElement` function globally, true by defaultIf `registerGlobal` is false you can use eachElement from the plugin:```js
* `fn` **[Function][7]** Returns **([Promise][8]\<any> | [undefined][9])** ## fakerTransformUse the `@faker-js/faker` package to generate fake data inside examples on your gherkin tests#### UsageTo start please install `@faker-js/faker` package npm install -D @faker-js/faker yarn add -D @faker-js/fakerAdd this plugin to config file:```js
426
+
* `fn` **[Function][8]** Returns **([Promise][9]\<any> | [undefined][10])** ## fakerTransformUse the `@faker-js/faker` package to generate fake data inside examples on your gherkin tests#### UsageTo start please install `@faker-js/faker` package npm install -D @faker-js/faker yarn add -D @faker-js/fakerAdd this plugin to config file:```js
390
427
plugins: {
391
428
fakerTransform: {
392
429
enabled: true
@@ -400,7 +437,7 @@ Scenario Outline: ...
400
437
Examples:
401
438
| productName | customer | email | anythingMore |
402
439
| {{commerce.product}} | Dr. {{name.findName}} | {{internet.email}} | staticData |
403
-
```### Parameters* `config`  ## healSelf-healing tests with AI.Read more about heaking in [Self-Healing Tests][10]```js
440
+
```### Parameters* `config`  ## healSelf-healing tests with AI.Read more about heaking in [Self-Healing Tests][11]```js
404
441
plugins: {
405
442
heal: {
406
443
enabled: true,
@@ -414,7 +451,7 @@ plugins: {
414
451
enabled: true,
415
452
}
416
453
```Additional config options:* `errorClasses` - list of classes to search for errors (default: `['error', 'warning', 'alert', 'danger']`)
417
-
* `browserLogs` - list of types of errors to search for in browser logs (default: `['error']`)### Parameters* `config` (optional, default `{}`)## pauseOnFailAutomatically launches [interactive pause][11] when a test fails.Useful for debugging flaky tests on local environment.
454
+
* `browserLogs` - list of types of errors to search for in browser logs (default: `['error']`)### Parameters* `config` (optional, default `{}`)## pauseOnFailAutomatically launches [interactive pause][12] when a test fails.Useful for debugging flaky tests on local environment.
418
455
Add this plugin to config file:```js
419
456
plugins: {
420
457
pauseOnFail: {},
@@ -462,8 +499,8 @@ plugins: {
462
499
}
463
500
}
464
501
```Possible config options:* `uniqueScreenshotNames`: use unique names for screenshot. Default: false.
465
-
* `fullPageScreenshots`: make full page screenshots. Default: false.### Parameters* `config`  ## selenoid[Selenoid][12] plugin automatically starts browsers and video recording.
466
-
Works with WebDriver helper.### PrerequisiteThis plugin **requires Docker** to be installed.> If you have issues starting Selenoid with this plugin consider using the official [Configuration Manager][13] tool from Selenoid### UsageSelenoid plugin can be started in two ways:1. **Automatic** - this plugin will create and manage selenoid container for you.
502
+
* `fullPageScreenshots`: make full page screenshots. Default: false.### Parameters* `config`  ## selenoid[Selenoid][13] plugin automatically starts browsers and video recording.
503
+
Works with WebDriver helper.### PrerequisiteThis plugin **requires Docker** to be installed.> If you have issues starting Selenoid with this plugin consider using the official [Configuration Manager][14] tool from Selenoid### UsageSelenoid plugin can be started in two ways:1. **Automatic** - this plugin will create and manage selenoid container for you.
467
504
2. **Manual** - you create the conatainer and configure it with a plugin (recommended).#### AutomaticIf you are new to Selenoid and you want plug and play setup use automatic mode.Add plugin configuration in `codecept.conf.js`:```js
468
505
plugins: {
469
506
selenoid: {
@@ -476,10 +513,10 @@ plugins: {
476
513
enableLog: true,
477
514
},
478
515
}
479
-
```When `autoCreate` is enabled it will pull the [latest Selenoid from DockerHub][14] and start Selenoid automatically.
516
+
```When `autoCreate` is enabled it will pull the [latest Selenoid from DockerHub][15] and start Selenoid automatically.
480
517
It will also create `browsers.json` file required by Selenoid.In automatic mode the latest version of browser will be used for tests. It is recommended to specify exact version of each browser inside `browsers.json` file.> **If you are using Windows machine or if `autoCreate` does not work properly, create container manually**#### ManualWhile this plugin can create containers for you for better control it is recommended to create and launch containers manually.
481
-
This is especially useful for Continous Integration server as you can configure scaling for Selenoid containers.> Use [Selenoid Configuration Manager][13] to create and start containers semi-automatically.1. Create `browsers.json` file in the same directory `codecept.conf.js` is located
482
-
[Refer to Selenoid documentation][15] to know more about browsers.json.*Sample browsers.json*```js
518
+
This is especially useful for Continous Integration server as you can configure scaling for Selenoid containers.> Use [Selenoid Configuration Manager][14] to create and start containers semi-automatically.1. Create `browsers.json` file in the same directory `codecept.conf.js` is located
519
+
[Refer to Selenoid documentation][16] to know more about browsers.json.*Sample browsers.json*```js
483
520
{
484
521
"chrome": {
485
522
"default": "latest",
@@ -492,7 +529,7 @@ This is especially useful for Continous Integration server as you can configure
492
529
}
493
530
}
494
531
}
495
-
```> It is recommended to use specific versions of browsers in `browsers.json` instead of latest. This will prevent tests fail when browsers will be updated.**⚠ At first launch selenoid plugin takes extra time to download all Docker images before tests starts**.2. Create Selenoid containerRun the following command to create a container. To know more [refer here][16]```bash
532
+
```> It is recommended to use specific versions of browsers in `browsers.json` instead of latest. This will prevent tests fail when browsers will be updated.**⚠ At first launch selenoid plugin takes extra time to download all Docker images before tests starts**.2. Create Selenoid containerRun the following command to create a container. To know more [refer here][17]```bash
496
533
docker create \
497
534
--name selenoid \
498
535
-p 4444:4444 \
@@ -511,7 +548,7 @@ To save space videos for all succesful tests are deleted. This can be changed by
511
548
| enableVideo | Enable video recording and use `video` folder of output (default: false) |
512
549
| enableLog | Enable log recording and use `logs` folder of output (default: false) |
513
550
| deletePassed | Delete video and logs of passed tests (default : true) |
514
-
| additionalParams | example: `additionalParams: '--env TEST=test'` [Refer here][17] to know more |### Parameters* `config`  ## stepByStepReport![step-by-step-report][18]Generates step by step report for a test.
551
+
| additionalParams | example: `additionalParams: '--env TEST=test'` [Refer here][18] to know more |### Parameters* `config`  ## stepByStepReport![step-by-step-report][19]Generates step by step report for a test.
515
552
After each step in a test a screenshot is created. After test executed screenshots are combined into slideshow.
516
553
By default, reports are generated only for failed tests.Run tests with plugin enabled: npx codeceptjs run --plugins stepByStepReport#### Configuration```js
517
554
"plugins": {
@@ -568,9 +605,9 @@ plugins: {
568
605
* sauce
569
606
* testingbot
570
607
* browserstack
571
-
* appiumA complete list of all available services can be found on [webdriverio website][19].#### Setup1. Install a webdriverio service
608
+
* appiumA complete list of all available services can be found on [webdriverio website][20].#### Setup1. Install a webdriverio service
572
609
2. Enable `wdio` plugin in config
573
-
3. Add service name to `services` array inside wdio plugin config.See examples below:#### Selenium Standalone ServiceInstall ` @wdio/selenium-standalone-service` package, as [described here][20].
610
+
3. Add service name to `services` array inside wdio plugin config.See examples below:#### Selenium Standalone ServiceInstall ` @wdio/selenium-standalone-service` package, as [described here][21].
574
611
It is important to make sure it is compatible with current webdriverio version.Enable `wdio` plugin in plugins list and add `selenium-standalone` service:```js
575
612
plugins: {
576
613
wdio: {
@@ -579,7 +616,7 @@ plugins: {
579
616
// additional config for service can be passed here
580
617
}
581
618
}
582
-
```#### Sauce ServiceInstall `@wdio/sauce-service` package, as [described here][21].
619
+
```#### Sauce ServiceInstall `@wdio/sauce-service` package, as [described here][22].
583
620
It is important to make sure it is compatible with current webdriverio version.Enable `wdio` plugin in plugins list and add `sauce` service:```js
584
621
plugins: {
585
622
wdio: {
@@ -591,5 +628,5 @@ plugins: {
591
628
}
592
629
}
593
630
```***In the same manner additional services from webdriverio can be installed, enabled, and configured.#### Configuration* `services` - list of enabled services
0 commit comments