Skip to content

Commit 4f7a936

Browse files
CopilotCopilot
authored
[WIP] Retries mechanism improvements (#5103)
1 parent 73960e8 commit 4f7a936

File tree

6 files changed

+1130
-0
lines changed

6 files changed

+1130
-0
lines changed

‎docs/retry-mechanisms-enhancement.md

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
# Retry Mechanisms Enhancement
2+
3+
This document describes the improvements made to CodeceptJS retry mechanisms to eliminate overlaps and provide better coordination.
4+
5+
## Problem Statement
6+
7+
CodeceptJS previously had multiple retry mechanisms that could overlap and conflict:
8+
9+
1. **Global Retry Configuration** - Feature and Scenario level retries
10+
2. **RetryFailedStep Plugin** - Individual step retries
11+
3. **Manual Step Retries** - `I.retry()` calls
12+
4. **Hook Retries** - Before/After hook retries
13+
5. **Helper Retries** - Helper-specific retry mechanisms
14+
15+
These mechanisms could result in:
16+
17+
- Exponential retry counts (e.g., 3 scenario retries × 2 step retries = 6 total executions per step)
18+
- Conflicting configurations with no clear precedence
19+
- Confusing logging and unclear behavior
20+
- Difficult debugging when multiple retry levels were active
21+
22+
## Solution Overview
23+
24+
### 1. Enhanced Global Retry (`lib/listener/enhancedGlobalRetry.js`)
25+
26+
**New Features:**
27+
28+
- Priority-based retry coordination
29+
- Clear precedence system
30+
- Enhanced logging with mechanism identification
31+
- Backward compatibility with existing configurations
32+
33+
**Priority System:**
34+
35+
```javascript
36+
const RETRY_PRIORITIES = {
37+
MANUAL_STEP: 100, // I.retry() or step.retry() - highest priority
38+
STEP_PLUGIN: 50, // retryFailedStep plugin
39+
SCENARIO_CONFIG: 30, // Global scenario retry config
40+
FEATURE_CONFIG: 20, // Global feature retry config
41+
HOOK_CONFIG: 10, // Hook retry config - lowest priority
42+
}
43+
```
44+
45+
### 2. Enhanced RetryFailedStep Plugin (`lib/plugin/enhancedRetryFailedStep.js`)
46+
47+
**New Features:**
48+
49+
- Automatic coordination with scenario-level retries
50+
- Smart deferral when scenario retries are configured
51+
- Priority-aware retry registration
52+
- Improved logging and debugging information
53+
54+
**Coordination Logic:**
55+
56+
- When scenario retries are configured, step retries are automatically deferred to avoid excessive retry counts
57+
- Users can override this with `deferToScenarioRetries: false`
58+
- Clear logging explains coordination decisions
59+
60+
### 3. Retry Coordinator (`lib/retryCoordinator.js`)
61+
62+
**New Features:**
63+
64+
- Central coordination service for all retry mechanisms
65+
- Configuration validation with warnings for potential conflicts
66+
- Retry registration and priority management
67+
- Summary reporting for debugging
68+
69+
**Key Functions:**
70+
71+
- `validateConfig()` - Detects configuration conflicts and excessive retry counts
72+
- `registerRetry()` - Registers retry mechanisms with priority coordination
73+
- `getEffectiveRetryConfig()` - Returns the active retry configuration for a target
74+
- `generateRetrySummary()` - Provides debugging information about active retry mechanisms
75+
76+
## Migration Guide
77+
78+
### Immediate Benefits (No Changes Needed)
79+
80+
The enhanced retry mechanisms are **backward compatible**. Existing configurations will continue to work with these improvements:
81+
82+
- Better coordination between retry mechanisms
83+
- Enhanced logging for debugging
84+
- Automatic conflict detection and resolution
85+
86+
### Recommended Configuration Updates
87+
88+
#### 1. For Simple Cases - Use Scenario Retries Only
89+
90+
**Old Configuration (potentially conflicting):**
91+
92+
```javascript
93+
module.exports = {
94+
retry: 3, // scenario retries
95+
plugins: {
96+
retryFailedStep: {
97+
enabled: true,
98+
retries: 2, // step retries - could result in 3 * 3 = 9 executions
99+
},
100+
},
101+
}
102+
```
103+
104+
**Recommended Configuration:**
105+
106+
```javascript
107+
module.exports = {
108+
retry: 3, // scenario retries only
109+
plugins: {
110+
retryFailedStep: {
111+
enabled: false, // disable to avoid conflicts
112+
},
113+
},
114+
}
115+
```
116+
117+
#### 2. For Step-Level Control - Use Step Retries Only
118+
119+
**Recommended Configuration:**
120+
121+
```javascript
122+
module.exports = {
123+
plugins: {
124+
retryFailedStep: {
125+
enabled: true,
126+
retries: 2,
127+
ignoredSteps: ['amOnPage', 'wait*'], // customize as needed
128+
},
129+
},
130+
// No global retry configuration
131+
}
132+
```
133+
134+
#### 3. For Mixed Scenarios - Use Enhanced Coordination
135+
136+
```javascript
137+
module.exports = {
138+
retry: {
139+
Scenario: 2, // scenario retries for most tests
140+
},
141+
plugins: {
142+
retryFailedStep: {
143+
enabled: true,
144+
retries: 1,
145+
deferToScenarioRetries: true, // automatically coordinate (default)
146+
},
147+
},
148+
}
149+
```
150+
151+
### Testing Your Configuration
152+
153+
Use the new retry coordinator to validate your configuration:
154+
155+
```javascript
156+
const retryCoordinator = require('codeceptjs/lib/retryCoordinator')
157+
158+
// Validate your configuration
159+
const warnings = retryCoordinator.validateConfig(yourConfig)
160+
if (warnings.length > 0) {
161+
console.log('Retry configuration warnings:')
162+
warnings.forEach(warning => console.log(' -', warning))
163+
}
164+
```
165+
166+
## Enhanced Logging
167+
168+
The new retry mechanisms provide clearer logging:
169+
170+
```
171+
[Global Retry] Scenario retries: 3
172+
[Step Retry] Deferred to scenario retries (3 retries)
173+
[Retry Coordinator] Registered scenario retry (priority: 30)
174+
```
175+
176+
## Breaking Changes
177+
178+
**None.** All existing configurations continue to work.
179+
180+
## New Configuration Options
181+
182+
### Enhanced RetryFailedStep Plugin
183+
184+
```javascript
185+
plugins: {
186+
retryFailedStep: {
187+
enabled: true,
188+
retries: 2,
189+
deferToScenarioRetries: true, // NEW: automatically coordinate with scenario retries
190+
minTimeout: 1000,
191+
maxTimeout: 10000,
192+
factor: 1.5,
193+
ignoredSteps: ['wait*', 'amOnPage']
194+
}
195+
}
196+
```
197+
198+
### New Options:
199+
200+
- `deferToScenarioRetries` (boolean, default: true) - When true, step retries are disabled if scenario retries are configured
201+
202+
## Debugging Retry Issues
203+
204+
### 1. Check Configuration Validation
205+
206+
```javascript
207+
const retryCoordinator = require('codeceptjs/lib/retryCoordinator')
208+
const warnings = retryCoordinator.validateConfig(Config.get())
209+
console.log('Configuration warnings:', warnings)
210+
```
211+
212+
### 2. Monitor Enhanced Logging
213+
214+
Run tests with `--verbose` to see detailed retry coordination logs.
215+
216+
### 3. Generate Retry Summary
217+
218+
```javascript
219+
// In your test hooks
220+
const summary = retryCoordinator.generateRetrySummary()
221+
console.log('Retry mechanisms active:', summary)
222+
```
223+
224+
## Best Practices
225+
226+
1. **Choose One Primary Retry Strategy** - Either scenario-level OR step-level retries, not both
227+
2. **Use Configuration Validation** - Check for conflicts before running tests
228+
3. **Monitor Retry Logs** - Use enhanced logging to understand retry behavior
229+
4. **Test Retry Behavior** - Verify your retry configuration works as expected
230+
5. **Avoid Excessive Retries** - High retry counts often indicate test stability issues
231+
232+
## Future Enhancements
233+
234+
- Integration with retry coordinator for all retry mechanisms
235+
- Runtime retry strategy adjustment
236+
- Retry analytics and reporting
237+
- Advanced retry patterns (exponential backoff, conditional retries)

‎lib/listener/enhancedGlobalRetry.js

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
const event = require('../event')
2+
const output = require('../output')
3+
const Config = require('../config')
4+
const { isNotSet } = require('../utils')
5+
6+
const hooks = ['Before', 'After', 'BeforeSuite', 'AfterSuite']
7+
8+
/**
9+
* Priority levels for retry mechanisms (higher number = higher priority)
10+
* This ensures consistent behavior when multiple retry mechanisms are active
11+
*/
12+
const RETRY_PRIORITIES = {
13+
MANUAL_STEP: 100, // I.retry() or step.retry() - highest priority
14+
STEP_PLUGIN: 50, // retryFailedStep plugin
15+
SCENARIO_CONFIG: 30, // Global scenario retry config
16+
FEATURE_CONFIG: 20, // Global feature retry config
17+
HOOK_CONFIG: 10, // Hook retry config - lowest priority
18+
}
19+
20+
/**
21+
* Enhanced global retry mechanism that coordinates with other retry types
22+
*/
23+
module.exports = function () {
24+
event.dispatcher.on(event.suite.before, suite => {
25+
let retryConfig = Config.get('retry')
26+
if (!retryConfig) return
27+
28+
if (Number.isInteger(+retryConfig)) {
29+
// is number - apply as feature-level retry
30+
const retryNum = +retryConfig
31+
output.log(`[Global Retry] Feature retries: ${retryNum}`)
32+
33+
// Only set if not already set by higher priority mechanism
34+
if (isNotSet(suite.retries())) {
35+
suite.retries(retryNum)
36+
suite.opts.retryPriority = RETRY_PRIORITIES.FEATURE_CONFIG
37+
}
38+
return
39+
}
40+
41+
if (!Array.isArray(retryConfig)) {
42+
retryConfig = [retryConfig]
43+
}
44+
45+
for (const config of retryConfig) {
46+
if (config.grep) {
47+
if (!suite.title.includes(config.grep)) continue
48+
}
49+
50+
// Handle hook retries with priority awareness
51+
hooks
52+
.filter(hook => !!config[hook])
53+
.forEach(hook => {
54+
const retryKey = `retry${hook}`
55+
if (isNotSet(suite.opts[retryKey])) {
56+
suite.opts[retryKey] = config[hook]
57+
suite.opts[`${retryKey}Priority`] = RETRY_PRIORITIES.HOOK_CONFIG
58+
}
59+
})
60+
61+
// Handle feature-level retries
62+
if (config.Feature) {
63+
if (isNotSet(suite.retries()) || (suite.opts.retryPriority || 0) <= RETRY_PRIORITIES.FEATURE_CONFIG) {
64+
suite.retries(config.Feature)
65+
suite.opts.retryPriority = RETRY_PRIORITIES.FEATURE_CONFIG
66+
output.log(`[Global Retry] Feature retries: ${config.Feature}`)
67+
}
68+
}
69+
}
70+
})
71+
72+
event.dispatcher.on(event.test.before, test => {
73+
let retryConfig = Config.get('retry')
74+
if (!retryConfig) return
75+
76+
if (Number.isInteger(+retryConfig)) {
77+
// Only set if not already set by higher priority mechanism
78+
if (test.retries() === -1) {
79+
test.retries(retryConfig)
80+
test.opts.retryPriority = RETRY_PRIORITIES.SCENARIO_CONFIG
81+
output.log(`[Global Retry] Scenario retries: ${retryConfig}`)
82+
}
83+
return
84+
}
85+
86+
if (!Array.isArray(retryConfig)) {
87+
retryConfig = [retryConfig]
88+
}
89+
90+
retryConfig = retryConfig.filter(config => !!config.Scenario)
91+
92+
for (const config of retryConfig) {
93+
if (config.grep) {
94+
if (!test.fullTitle().includes(config.grep)) continue
95+
}
96+
97+
if (config.Scenario) {
98+
// Respect priority system
99+
if (test.retries() === -1 || (test.opts.retryPriority || 0) <= RETRY_PRIORITIES.SCENARIO_CONFIG) {
100+
test.retries(config.Scenario)
101+
test.opts.retryPriority = RETRY_PRIORITIES.SCENARIO_CONFIG
102+
output.log(`[Global Retry] Scenario retries: ${config.Scenario}`)
103+
}
104+
}
105+
}
106+
})
107+
}
108+
109+
// Export priority constants for use by other retry mechanisms
110+
module.exports.RETRY_PRIORITIES = RETRY_PRIORITIES

0 commit comments

Comments
 (0)