updated docs for AI and effects

DavertMik · DavertMik · commit 851e31a7c793 · 2025-02-10T05:16:26.000+02:00
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,9 +4,9 @@ This release introduces major new features and internal refactoring. It is an im
 
 🛩️ _Features_
 
-### 🔥 **[Element functions](./els)**
+### 🔥 **Native Element Functions**
 
-A new API for direct element interactions has been introduced. This API provides low-level element manipulation functions for more granular control over element interactions and assertions:
+A new [Els API](./els) for direct element interactions has been introduced. This API provides low-level element manipulation functions for more granular control over element interactions and assertions:
 
 - `element()` - perform custom operations on first matching element
 - `eachElement()` - iterate and perform operations on each matching element
@@ -49,7 +49,7 @@ Scenario('element functions demo', async ({ I }) => {
 
 ### 🔮 **Effects introduced**
 
-Effects is a new concept that encompasses all functions that can modify scenario flow. These functions are now part of a single module. Previously, they were used via plugins like `tryTo` and `retryTo`. Now, it is recommended to import them directly:
+[Effects](./effects) is a new concept that encompasses all functions that can modify scenario flow. These functions are now part of a single module. Previously, they were used via plugins like `tryTo` and `retryTo`. Now, it is recommended to import them directly:
 
 ```js
 const { tryTo, retryTo } = require('codeceptjs/effects')
diff --git a/docs/ai.md b/docs/ai.md
@@ -85,7 +85,7 @@ ai: {
     const openai = new OpenAI({ apiKey: process.env['OPENAI_API_KEY'] })
 
     const completion = await openai.chat.completions.create({
-      model: 'gpt-3.5-turbo-0125',
+      model: 'gpt-3.5-turbo',
       messages,
     })
 
@@ -354,6 +354,131 @@ npx codeceptjs run --ai
 When execution finishes, you will receive information on token usage and code suggestions proposed by AI.
 By evaluating this information you will be able to check how effective AI can be for your case.
 
+## Analyze Results
+
+When running tests with AI enabled, CodeceptJS can automatically analyze test failures and provide insights. The analyze plugin helps identify patterns in test failures and provides detailed explanations of what went wrong.
+
+Enable the analyze plugin in your config:
+
+```js
+plugins: {
+  analyze: {
+    enabled: true,
+    // analyze up to 3 failures in detail
+    analyze: 3,
+    // group similar failures when 5 or more tests fail
+    clusterize: 5,
+    // enable screenshot analysis (requires modal that can analyze  screenshots)
+    vision: false
+  }
+}
+```
+
+When tests are executed with `--ai` flag, the analyze plugin will:
+
+**Analyze Individual Failures**: For each failed test (up to the `analyze` limit), it will:
+
+- Examine the error message and stack trace
+- Review the test steps that led to the failure
+- Provide a detailed explanation of what likely caused the failure
+- Suggest possible fixes and improvements
+
+Sample Analysis report:
+
+When analyzing individual failures (less than `clusterize` threshold), the output looks like this:
+
+```
+🪄 AI REPORT:
+--------------------------------
+→ Cannot submit registration form with invalid email 👀
+
+* SUMMARY: Form submission failed due to invalid email format, system correctly shows validation message
+* ERROR: expected element ".success-message" to be visible, but it is not present in DOM
+* CATEGORY: Data errors (password incorrect, no options in select, invalid format, etc)
+* STEPS: I.fillField('#email', 'invalid-email'); I.click('Submit'); I.see('.success-message')
+* URL: /register
+
+```
+
+> The 👀 emoji indicates that screenshot analysis was performed (when `vision: true`).
+
+**Cluster Similar Failures**: When number of failures exceeds the `clusterize` threshold:
+
+- Groups failures with similar error patterns
+- Identifies common root causes
+- Suggests fixes that could resolve multiple failures
+- Helps prioritize which issues to tackle first
+
+**Categorize Failures**: Automatically classifies failures into categories like:
+
+- Browser/connection issues
+- Network errors
+- Element locator problems
+- Navigation errors
+- Code errors
+- Data validation issues
+- etc.
+
+Clusterization output:
+
+```
+🪄 AI REPORT:
+_______________________________
+
+## Group 1 🔍
+
+* SUMMARY: Element locator failures across login flow
+* CATEGORY: HTML / page elements (not found, not visible, etc)
+* ERROR: Element "#login-button" is not visible
+* STEP: I.click('#login-button')
+* SUITE: Authentication
+* TAG: @login
+* AFFECTED TESTS (4):
+    x Cannot login with valid credentials
+    x Should show error on invalid login
+    x Login button should be disabled when form empty
+    x Should redirect to dashboard after login
+
+## Group 2 🌐
+
+* SUMMARY: API timeout issues during user data fetch
+* CATEGORY: Network errors (server error, timeout, etc)
+* URL: /api/v1/users
+* ERROR: Request failed with status code 504, Gateway Timeout
+* SUITE: User Management
+* AFFECTED TESTS (3):
+    x Should load user profile data
+    x Should display user settings
+    x Should fetch user notifications
+
+## Group 3 ⚠️
+
+* SUMMARY: Form validation errors on registration page
+* CATEGORY: Data errors (password incorrect, no options in select, invalid format, etc)
+* ERROR: Expected field "password" to have error "Must be at least 8 characters"
+* STEP: I.see('Must be at least 8 characters', '.error-message')
+* SUITE: User Registration
+* TAG: @registration
+* AFFECTED TESTS (2):
+    x Should validate password requirements
+    x Should show all validation errors on submit
+```
+
+If `vision: true` is enabled and your tests take screenshots on failure, the plugin will also analyze screenshots to provide additional visual context about the failure.
+
+The analysis helps teams:
+
+- Quickly understand the root cause of failures
+- Identify patterns in failing tests
+- Prioritize fixes based on impact
+- Maintain more stable test suites
+
+Run tests with both AI and analyze enabled:
+
+```bash
+npx codeceptjs run --ai
+```
+
 ## Arbitrary Prompts
 
 What if you want to take AI on the journey of test automation and ask it questions while browsing pages?
diff --git a/docs/effects.md b/docs/effects.md
@@ -0,0 +1,101 @@
+# Effects
+
+Effects are functions that can modify scenario flow. They provide ways to handle conditional steps, retries, and test flow control.
+
+## Installation
+
+Effects can be imported directly from CodeceptJS:
+
+```js
+const { tryTo, retryTo, within } = require('codeceptjs/effects')
+```
+
+> 📝 Note: Prior to v3.7, `tryTo` and `retryTo` were available globally via plugins. This behavior is deprecated and will be removed in v4.0.
+
+## tryTo
+
+The `tryTo` effect allows you to attempt steps that may fail without stopping test execution. It's useful for handling optional steps or conditions that aren't critical for the test flow.
+
+```js
+const { tryTo } = require('codeceptjs/effects')
+
+// inside a test
+const success = await tryTo(() => {
+  // These steps may fail but won't stop the test
+  I.see('Cookie banner')
+  I.click('Accept cookies')
+})
+
+if (!success) {
+  I.say('Cookie banner was not found')
+}
+```
+
+If the steps inside `tryTo` fail:
+
+- The test will continue execution
+- The failure will be logged in debug output
+- `tryTo` returns `false`
+- Auto-retries are disabled inside `tryTo` blocks
+
+## retryTo
+
+The `retryTo` effect allows you to retry a set of steps multiple times until they succeed. This is useful for handling flaky elements or conditions that may need multiple attempts.
+
+```js
+const { retryTo } = require('codeceptjs/effects')
+
+// Retry up to 5 times with 200ms between attempts
+await retryTo(() => {
+  I.switchTo('#editor-frame')
+  I.fillField('textarea', 'Hello world')
+}, 5)
+```
+
+Parameters:
+
+- `callback` - Function containing steps to retry
+- `maxTries` - Maximum number of retry attempts
+- `pollInterval` - (optional) Delay between retries in milliseconds (default: 200ms)
+
+The callback receives the current retry count as an argument:
+
+```js
+const { retryTo } = require('codeceptjs/effects')
+
+// inside a test...
+await retryTo(tries => {
+  I.say(`Attempt ${tries}`)
+  I.click('Submit')
+  I.see('Success')
+}, 3)
+```
+
+## within
+
+The `within` effect allows you to perform multiple steps within a specific context (like an iframe or modal):
+
+```js
+const { within } = require('codeceptjs/effects')
+
+// inside a test...
+
+within('.modal', () => {
+  I.see('Modal title')
+  I.click('Close')
+})
+```
+
+## Usage with TypeScript
+
+Effects are fully typed and work well with TypeScript:
+
+```ts
+import { tryTo, retryTo, within } from 'codeceptjs/effects'
+
+const success = await tryTo(async () => {
+  await I.see('Element')
+})
+```
+
+This documentation covers the main effects functionality while providing practical examples and important notes about deprecation and future changes. Let me know if you'd like me to expand any section or add more examples!
diff --git a/lib/plugin/analyze.js b/lib/plugin/analyze.js
@@ -60,7 +60,7 @@ const defaultConfig = {
 
         If there is no groups of tests, say: "No patterns found"
         Preserve error messages but cut them if they are too long.
-        Respond clearly and directly, without introductory words or phrases like ‘Of course,’ ‘Here is the answer,’ etc.
+        Respond clearly and directly, without introductory words or phrases like 'Of course,' 'Here is the answer,' etc.
         Do not list more than 3 errors in the group.
         If you identify that all tests in the group have the same tag, add this tag to the group report, otherwise ignore TAG section.
         If you identify that all tests in the group have the same suite, add this suite to the group report, otherwise ignore SUITE section.
@@ -160,9 +160,56 @@ const defaultConfig = {
 }
 
 /**
+ * CodeceptJS Analyze Plugin - Uses AI to analyze test failures and provide insights
  *
- * @param {*} config
- * @returns
+ * This plugin analyzes failed tests using AI to provide detailed explanations and group similar failures.
+ * When enabled with --ai flag, it generates reports after test execution.
+ *
+ * #### Usage
+ *
+ * ```js
+ * // in codecept.conf.js
+ * exports.config = {
+ *   plugins: {
+ *     analyze: {
+ *       enabled: true,
+ *       clusterize: 5,
+ *       analyze: 2,
+ *       vision: false
+ *     }
+ *   }
+ * }
+ * ```
+ *
+ * #### Configuration
+ *
+ * * `clusterize` (number) - minimum number of failures to trigger clustering analysis. Default: 5
+ * * `analyze` (number) - maximum number of individual test failures to analyze in detail. Default: 2
+ * * `vision` (boolean) - enables visual analysis of test screenshots. Default: false
+ * * `categories` (array) - list of failure categories for classification. Defaults to:
+ *   - Browser connection error / browser crash
+ *   - Network errors (server error, timeout, etc)
+ *   - HTML / page elements (not found, not visible, etc)
+ *   - Navigation errors (404, etc)
+ *   - Code errors (syntax error, JS errors, etc)
+ *   - Library & framework errors
+ *   - Data errors (password incorrect, invalid format, etc)
+ *   - Assertion failures
+ *   - Other errors
+ * * `prompts` (object) - customize AI prompts for analysis
+ *   - `clusterize` - prompt for clustering analysis
+ *   - `analyze` - prompt for individual test analysis
+ *
+ * #### Features
+ *
+ * * Groups similar failures when number of failures >= clusterize value
+ * * Provides detailed analysis of individual failures
+ * * Analyzes screenshots if vision=true and screenshots are available
+ * * Classifies failures into predefined categories
+ * * Suggests possible causes and solutions
+ *
+ * @param {Object} config - Plugin configuration
+ * @returns {void}
  */
 module.exports = function (config = {}) {
   config = Object.assign(defaultConfig, config)
