You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Command Logs Analytics uses the WebDriver command logs to analyze your failed commands or steps. The WebDriver commands logs are the steps which are executed by the test script during the test execution. To know more about the WebDriver command logs, you can refer to the [W3C Command Logs](https://www.w3.org/TR/webdriver2/#endpoints).
To read more about the Command Logs Analytics, you can refer to the [documentation](/docs/analytics-modules-test-intelligence-command-logs-analytics/) here.
Copy file name to clipboardExpand all lines: docs/test-intelligence-ftd.md
+54-34Lines changed: 54 additions & 34 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -63,25 +63,57 @@ There are following Prerequisites to use Flaky Test Detection:
63
63
:::info
64
64
### What are Flaky Tests?
65
65
66
-
Flaky tests, refer to tests that exhibit inconsistent behavior when executed multiple times under the same conditions. These tests may produce varying results (pass or fail) when run repeatedly, even without any changes to the application or test script.
66
+
Flaky tests refer to tests that exhibit inconsistent behavior when executed multiple times under the same conditions. These tests may produce varying results (pass or fail) when run repeatedly, even without any changes to the application or test script.
67
67
:::
68
68
69
-
### Selenium Tests of Automation and HyperExecute
69
+
### Flaky Test Detection Logics Supported
70
70
71
-
Flaky Test Detection uses the WebDriver command logs to identify the flaky <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> tests. The WebDriver commands logs are the steps which are executed by the test script during the test execution. To know more about the WebDriver command logs, you can refer to the [W3C Command Logs](https://www.w3.org/TR/webdriver2/#endpoints).
71
+
**Summary:**
72
72
73
-
It uses the following algorithm to identify the flaky tests:
73
+
Flaky Test Detection on the platform supports two complementary methods for identifying genuinely flaky tests. However, only one logic can be active at a time, based on user selection in the configuration. The detection of flaky tests will be performed exclusively using the selected logic:
74
74
75
-
- Tests marked as Flaky <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> have shown inconsistent behavior. The inconsistent behaviour is characterized by the transitions in the `status` of the test considered in the [sliding window of tests](/support/docs/test-intelligence-flakiness-test-detection/#how-to-use-the-custom-flaky-test-detection-settings).
76
-
- We utilize our in-house algorithm to map transitioning tests in accordance to the configurations specified by the user. A `test transition` occurs when the status of a test switches from Passed->Failed or Failed->Passed
77
-
- For instance, when analyzing a test's status sequence, Passed -> Failed -> Passed, signifies that the test's status has undergone two transitions within the most recent three executions and the transitions are spotted accordingly.
78
-
- Similarly the algorithm identifies the tests where the commands are showing a different behavior than the previous runs. This analysis examines individual test commands and their statuses, offering a granular view of potential sources of flakiness <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> at the command level.
75
+
1.**Command Logs Mapping:**
76
+
- Detects flakiness by analyzing transitions in test status and mapping command logs across test runs.
77
+
- Useful for identifying inconsistencies at the command level, such as when specific steps within a test behave differently across executions.
79
78
79
+
2.**Error Message Comparison:**
80
+
- Detects flakiness by comparing error messages (custom remark messages set by the user) between consecutive test runs.
81
+
- A test is classified as flaky only if it fails with *different* error messages across runs. If a test fails consistently with the *same* error message, it is not considered flaky.
82
+
- Reduces false positives by focusing on genuinely unstable tests.
80
83
81
-
84
+
**Configuration:**
85
+
- Users can select either Command Logs Mapping or Error Message Comparison as the active logic for flaky test detection in their settings.
86
+
- Only the selected logic will be applied for the detection of flaky tests. The other logic will not be considered until selected.
87
+
88
+
**How it works:**
89
+
90
+
- The system captures and stores both command logs and error messages from each test run.
91
+
- Flakiness is detected using only the logic chosen by the user:
92
+
- If Command Logs Mapping is selected, only command log inconsistencies are analyzed.
93
+
- If Error Message Comparison is selected, only error message differences are analyzed.
94
+
- Tests are marked as flaky if the selected method identifies inconsistent behavior or error messages.
95
+
96
+
**Examples:**
97
+
- If Command Logs Mapping is active: A test's command logs show different outcomes for the same command across runs → flaky.
98
+
- If Error Message Comparison is active: A test fails with "TimeoutError" in one run and "ElementNotFound" in another → flaky.
99
+
- If a test fails with the same error message and consistent command logs, it is not considered flaky under either logic.
100
+
101
+
**Technical Details:**
102
+
- The test result processing pipeline extracts and stores both command logs and error messages for each test run.
103
+
- Flakiness detection logic applies only the selected method.
104
+
- The database schema supports storage for both command logs and error messages.
105
+
- Logging is added to track the decision-making process for the active detection method.
106
+
- The UI reflects the detection mechanism, showing command log history or error message history based on the selected logic.
107
+
- Backward compatibility is maintained with existing test history data.
108
+
109
+
**Business Value:**
110
+
- Offers flexibility and precision in flaky test detection.
111
+
- Reduces false positives and helps teams focus on genuinely unstable tests.
112
+
- Improves the reliability of the test insights dashboard.
113
+
- Enables more targeted test stabilization efforts.
82
114
83
115
:::caution Supported Frameworks
84
-
Currently, this feature only support `Selenium` based tests for Web Automation and HyperExecute subscribers.
116
+
Currently, these features only support `Selenium` based tests for Web Automation and HyperExecute subscribers. Integration with Playwright reporter infrastructure is ensured. Support for additional frameworks is upcoming.
85
117
:::
86
118
87
119
@@ -97,31 +129,31 @@ To view the Flaky Test Detection report, you can follow the below steps:
## Exploring different metrics in Flaky Test Detection
132
+
## Exploring Different Metrics in Flaky Test Detection
101
133
102
-
In Flaky Test Detection , some metrics play a crucial role in assessing and categorizing test results. Understanding these metrics is essential for identifying and addressing inconsistencies in automated testing.
134
+
In Flaky Test Detection, several metrics play a crucial role in assessing and categorizing test results. Understanding these metrics is essential for identifying and addressing inconsistencies in automated testing. Please note that some metrics are only relevant when the **Command Logs Mapping** logic is selected.
103
135
104
136
1.**Flake Rate**
105
-
The Flake Rate is a metric that measures the current flake rate within a specific group of tests. In Flaky Test Detection, tests are grouped based on criteria such as test name, browser, operating system (OS), and resolution. This metric assesses the inconsistency across test runs of the same name by quantifying transitions in test status and provides valuable insights into the collective performance of similar tests under varying conditions.
137
+
The Flake Rate is a metric that measures the current flake rate within a specific group of tests. In Flaky Test Detection, tests are grouped based on criteria such as test name, browser, operating system (OS), and resolution. This metric assesses the inconsistency across test runs of the same name by quantifying transitions in test status and provides valuable insights into the collective performance of similar tests under varying conditions.
106
138
107
-
**Use Case:** Picture a group of tests. The Flake Rate serves as a measure of the degree of consistency or inconsistency observed in the test status, when these tests are executed repeatedly
139
+
**Use Case:** Picture a group of tests. The Flake Rate serves as a measure of the degree of consistency or inconsistency observed in the test status when these tests are executed repeatedly.
108
140
109
-
1.**Flaky Commands**
110
-
The No. of Flaky Commands is a metric used to define the order of flakiness <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> of an individual test. It signifies the inconsistency of different commands within the same test.
141
+
2.**Flaky Commands***(Available only with Command Logs Mapping logic)*
142
+
The Number of Flaky Commands is a metric used to define the order of flakiness <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> of an individual test. It signifies the inconsistency of different commands within the same test.
111
143
112
-
**Use Case:** Consider a single test that performs a series of commands. The number of flaky commands will let the user know how many commands have inconsistencies in a single test.
144
+
**Use Case:** Consider a single test that performs a series of commands. The number of flaky commands will let the user know how many commands have inconsistencies in a single test.
113
145
114
-
3.**Command Flakiness Percentage**
115
-
The Command Flakiness Percentage is a metric employed to define the order of flakiness <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> of a specific command within a test. It assesses the inconsistency of the same command executed across different test runs within the same group. This level of granularity helps pinpoint potential sources of flakiness <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> at the command level.
146
+
3.**Command Flakiness Percentage***(Available only with Command Logs Mapping logic)*
147
+
The Command Flakiness Percentage is a metric employed to define the order of flakiness <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> of a specific command within a test. It assesses the inconsistency of the same command executed across different test runs within the same group. This level of granularity helps pinpoint potential sources of flakiness <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> at the command level.
116
148
117
-
**Use Case:** Within a test, individual commands may produce inconsistent results. The Command Flakiness Percentage identifies the fluctuations in each command, aiding in precise root cause analysis.
149
+
**Use Case:** Within a test, individual commands may produce inconsistent results. The Command Flakiness Percentage identifies the fluctuations in each command, aiding in precise root cause analysis.
118
150
119
151
:::info
120
152
Understanding these metrics allows you to:
121
153
122
154
- Assess the overall consistency of tests within a group.
123
155
- Identify specific tests that exhibit inconsistent behavior.
124
-
- Drill down to individual commands to pinpoint sources of flakiness <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" />.
156
+
- Drill down to individual commands to pinpoint sources of flakiness <img src={require('../assets/images/test-intelligence/flake-icon.webp').default} alt="flake icon" width="16" /> (when Command Logs Mapping is active).
125
157
- Prioritize improvements and optimizations based on the level of inconsistency revealed by these metrics.
126
158
:::
127
159
@@ -173,16 +205,4 @@ Flakiness is characterized by the inconsistency observed in prior executions of
173
205
174
206
## Support for Cypress, Playwright, Puppeteer and Appium <NewTag value="UPCOMING" bgColor="#7c39ff" color="#fff" />
175
207
176
-
We are actively working on extending its Flaky Test Detection capabilities to include popular testing frameworks like `Cypress, Playwright, Puppeteer, and Appium`. This expansion will empower users to assess and manage test reliability across a broader spectrum of testing scenarios in both web and app based tests, enhancing the overall testing experience. Stay tuned for updates on the availability of Flaky Test Detection support for these frameworks.
177
-
178
-
:::note Join Waitlist
179
-
Are you looking interested in getting the Flaky Test Detection for your test suite?
We are actively working on extending its Flaky Test Detection capabilities to include popular testing frameworks like `Cypress, Playwright, Puppeteer, and Appium`. This expansion will empower users to assess and manage test reliability across a broader spectrum of testing scenarios in both web and app based tests, enhancing the overall testing experience. Stay tuned for updates on the availability of Flaky Test Detection support for these frameworks.
### Intelligent Flakiness Detection of WebDriver Command logs <NewTagvalue="BETA"bgColor="#ffec02"color="#000" />
48
+
### Intelligent Flakiness Detection of WebDriver Command logs
58
49
59
50
Experience the power of our AI algorithms that can detect inconsistent test results in your WebDriver command logs. With this feature, identifying flaky tests becomes a breeze, allowing you to focus on resolving issues effectively.
### Error Classification of Command Logs <NewTagvalue="UPCOMING"bgColor="#7c39ff"color="#fff" />
55
+
### Error Classification of Command Logs
65
56
66
57
Our platform goes beyond simple error detection by classifying different types of errors in your command logs. This classification enables you to prioritize and address issues accordingly, ensuring a more efficient and streamlined QA process.
Harness the capabilities of our advanced algorithms to identify trends in errors. By leveraging historical data and predictive analysis, <img loading="lazy" src="https://www.lambdatest.com/resources/images/testLogo.svg" alt="cmd" style={{width:"2%"}} className="doc_img"/> Test Intelligence helps you anticipate and prevent future issues, ultimately improving the overall quality of your applications.
71
62
72
-
### Anomalies in Test Execution Across Environments <NewTagvalue="UPCOMING"bgColor="#7c39ff"color="#fff" />
63
+
### Anomalies in Test Execution Across Environments
73
64
74
65
Detecting anomalies in test execution is crucial for ensuring consistency and reliability. Our platform excels in identifying and highlighting anomalies that occur across various environments, providing you with valuable insights to optimize your application's performance.
75
66
@@ -93,13 +84,4 @@ Ready to experience the power of <img loading="lazy" src="https://www.lambdatest
93
84
3. Explore the intuitive interface and dive into the automated analysis of your test data.
94
85
4. Discover valuable insights, address issues proactively, and optimize your testing process for better results.
95
86
96
-
Don't miss out on the opportunity to revolutionize your QA testing. Join us on this exciting journey with Test Intelligence and take your testing efforts to new heights. Get started today and unlock the full potential of your applications.
Don't miss out on the opportunity to revolutionize your QA testing. Join us on this exciting journey with Test Intelligence and take your testing efforts to new heights. Get started today and unlock the full potential of your applications.
0 commit comments