Skip to content

Commit b9cc770

Browse files
badeballbadeball
committed
Add an option for soft errors
In case of #1222 [1], the fail fast [2] strategy adopted here appeared to mask an underlying issue. I suspect this might also be the case for #1313 [3] and similar issues. Thus an option for erroring softly is introduced. With this option, instead of throwing fast in Cypress' event handlers, the preprocessor will emit an error message to stderr at the end of a run. My hope is that this will make it easier for the end-user to provide proper reproducible examples when running into these errors. [1] #1222 [2] https://martinfowler.com/ieeeSoftware/failFast.pdf [3] #1313
1 parent 9e4641b commit b9cc770

14 files changed

+1087
-646
lines changed

‎CHANGELOG.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,49 @@
22

33
All notable changes to this project will be documented in this file.
44

5+
## Unreleased
6+
7+
- Add a flag to turn expectation errors soft. Relates to [#1313](https://github.com/badeball/cypress-cucumber-preprocessor/issues/1313) and [#1222](https://github.com/badeball/cypress-cucumber-preprocessor/issues/1222).
8+
9+
- During creation of message reports, which the JSON & HTML reports are products of, some test state is tracked by this library. This requires taking into account a myriad of edge cases, mostly related to Cypress' reload behavior. There are however likely several remaining edge cases that are unaccounted for.
10+
11+
In case of [#1222](https://github.com/badeball/cypress-cucumber-preprocessor/issues/1222), the [fail fast](https://martinfowler.com/ieeeSoftware/failFast.pdf) strategy adopted here appeared to mask an underlying issue. I suspect this might also be the case for [#1313](https://github.com/badeball/cypress-cucumber-preprocessor/issues/1313) and similar issues. Thus an option for erroring softly is introduced. With this option, instead of throwing fast in Cypress' event handlers, the preprocessor will emit an error message to stderr at the end of a run.
12+
13+
My hope is that this will make it easier for the end-user to provide proper reproducible examples when running into these errors. Additionally, it can be useful for users whom which reports aren't strictly a necessity but do encounter such errors.
14+
15+
This option can be configured similar to other options, with examples shown below.
16+
17+
```
18+
// package.json
19+
{
20+
"dependencies": {
21+
"@badeball/cypress-cucumber-preprocessor": "latest"
22+
},
23+
"cypress-cucumber-preprocessor": {
24+
"state": {
25+
"softErrors": true
26+
}
27+
}
28+
}
29+
```
30+
31+
```
32+
// .cypress-cucumber-preprocessorrc.json
33+
{
34+
"state": {
35+
"softErrors": true
36+
}
37+
}
38+
```
39+
40+
```
41+
$ cypress run -e stateSoftErrors=true
42+
```
43+
44+
```
45+
$ env CYPRESS_stateSoftErrors=true cypress run
46+
```
47+
548
## v23.1.0
649
750
- Bumb engine requirement to Node v20.

‎docs/configuration.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ Every configuration option has a similar key which can be use to override it, sh
7474
| JSON path | Environment key | Example(s) |
7575
|------------------------------|-----------------------------|------------------------------------------|
7676
| `stepDefinitions` | `stepDefinitions` | `[filepath].{js,ts}` |
77+
| `state.softErrors` | `stateSoftErrors` | `true`, `false` |
7778
| `messages.enabled` | `messagesEnabled` | `true`, `false` |
7879
| `messages.output` | `messagesOutput` | `cucumber-messages.ndjson` |
7980
| `json.enabled` | `jsonEnabled` | `true`, `false` |

‎docs/faq.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,41 @@ You may have stumbled upon a configuration caveat (see [docs/configuration.md: C
8989

9090
You might be overriding some of the plugin's own event handlers (see [docs/event-handlers.md: On event handlers](https://github.com/badeball/cypress-cucumber-preprocessor/blob/master/docs/event-handlers.md)).
9191

92+
If that's not the case, you might've come across a bug in the preprocessor. If you intend to create a ticket, you <ins>have to</ins> provide a *reproducible example*. Sometimes it might not be clear what happens when these errors are produced, especially if they're intermittent.
93+
94+
In that case you can turn on "soft errors", which means that they *won't* halt the execution. This in turn *might* help you understand your edge case and thus help your provide a reproducible example. This option can be configured like shown below.
95+
96+
```
97+
// package.json
98+
{
99+
"dependencies": {
100+
"@badeball/cypress-cucumber-preprocessor": "latest"
101+
},
102+
"cypress-cucumber-preprocessor": {
103+
"state": {
104+
"softErrors": true
105+
}
106+
}
107+
}
108+
```
109+
110+
```
111+
// .cypress-cucumber-preprocessorrc.json
112+
{
113+
"state": {
114+
"softErrors": true
115+
}
116+
}
117+
```
118+
119+
```
120+
$ cypress run -e stateSoftErrors=true
121+
```
122+
123+
```
124+
$ env CYPRESS_stateSoftErrors=true cypress run
125+
```
126+
92127
<!-- Configuration issues -->
93128

94129
## I get `Webpack Compilation Error` (shown below)

‎features/soft_state_errors.feature‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
Feature: soft state errors
2+
3+
This essentially requires an edge case which this library does not account for. Currently, there's
4+
a single frontend error which messes with state tracking [1]. In order to simulate a backend
5+
error, a special flag is introduced to make the backend misbehave.
6+
7+
[1] https://github.com/badeball/cypress-cucumber-preprocessor/issues/1222#issuecomment-3056694748
8+
9+
Background:
10+
Given additional preprocessor configuration
11+
"""
12+
{
13+
"messages": {
14+
"enabled": true
15+
}
16+
}
17+
"""
18+
19+
Rule: it should fail hard by default upon state errors
20+
21+
Scenario: throw during Cypress' events
22+
Given a file named "cypress/e2e/a.feature" with:
23+
"""
24+
@env(origin="https://google.com/")
25+
Feature: a feature
26+
Scenario: a scenario
27+
Given a step
28+
"""
29+
And a file named "cypress/support/step_definitions/steps.js" with:
30+
"""
31+
const { Given } = require("@badeball/cypress-cucumber-preprocessor")
32+
Given("a step", function() {})
33+
// Inspired by https://github.com/badeball/cypress-cucumber-preprocessor/issues/1222#issuecomment-3056694748.
34+
Cypress.on("command:start", (c) => {
35+
if (typeof c.attributes.args[0] === "function") {
36+
throw "some error"
37+
}
38+
})
39+
"""
40+
When I run cypress
41+
Then it fails
42+
And the output should contain
43+
"""
44+
Expected there to be a timestamp for current step
45+
"""
46+
And there should be no messages report
47+
48+
Scenario: reload during before()
49+
Given a file named "cypress/e2e/a.feature" with:
50+
"""
51+
Feature: a feature
52+
Scenario: a scenario
53+
Given a step
54+
"""
55+
And a file named "cypress/support/step_definitions/steps.js" with:
56+
"""
57+
const { Given } = require("@badeball/cypress-cucumber-preprocessor")
58+
59+
before(() => {
60+
cy.visit("https://duckduckgo.com/");
61+
});
62+
63+
Given("a step", function() {})
64+
"""
65+
When I run cypress with "-e __cypress_cucumber_preprocessor_dont_use_this_simulate_backend_error=true"
66+
Then it fails
67+
And the output should contain
68+
"""
69+
Unexpected state in beforeSpecHandler: before-spec.
70+
"""
71+
And there should be no messages report
72+
73+
Scenario: throw during Cypress' events & soft error enabled
74+
Given additional preprocessor configuration
75+
"""
76+
{
77+
"state": {
78+
"softErrors": true
79+
}
80+
}
81+
"""
82+
And a file named "cypress/e2e/a.feature" with:
83+
"""
84+
Feature: a feature
85+
Scenario: a scenario
86+
Given a step
87+
"""
88+
And a file named "cypress/support/step_definitions/steps.js" with:
89+
"""
90+
const { Given } = require("@badeball/cypress-cucumber-preprocessor")
91+
Given("a step", function() {})
92+
// Inspired by https://github.com/badeball/cypress-cucumber-preprocessor/issues/1222#issuecomment-3056694748.
93+
Cypress.on("command:start", (c) => {
94+
if (typeof c.attributes.args[0] === "function") {
95+
throw "some error"
96+
}
97+
})
98+
"""
99+
When I run cypress
100+
Then it fails
101+
And stderr should contain
102+
"""
103+
A Cucumber library state error (shown bellow) occured in the frontend, thus no report is created.
104+
"""
105+
And there should be no messages report
106+
107+
Scenario: reload during before() & soft error enabled
108+
Given additional preprocessor configuration
109+
"""
110+
{
111+
"state": {
112+
"softErrors": true
113+
}
114+
}
115+
"""
116+
And a file named "cypress/e2e/a.feature" with:
117+
"""
118+
Feature: a feature
119+
Scenario: a scenario
120+
Given a step
121+
"""
122+
And a file named "cypress/support/step_definitions/steps.js" with:
123+
"""
124+
const { Given } = require("@badeball/cypress-cucumber-preprocessor")
125+
126+
before(() => {
127+
cy.visit("https://duckduckgo.com/");
128+
});
129+
130+
Given("a step", function() {})
131+
"""
132+
When I run cypress with "-e __cypress_cucumber_preprocessor_dont_use_this_simulate_backend_error=true"
133+
Then it passes
134+
And stderr should contain
135+
"""
136+
A Cucumber library state error (shown bellow) occured in the backend, thus no report is created.
137+
"""
138+
And there should be no messages report

‎features/step_definitions/cli_steps.ts‎

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -227,19 +227,30 @@ Then(
227227
},
228228
);
229229

230-
const normalizeOutput = (world: ICustomWorld) =>
231-
stripAnsi(expectLastRun(world).stdout)
232-
.replaceAll("\\", "/")
233-
.replaceAll("×", "✖");
230+
const normalizeOutput = (outout: string) =>
231+
stripAnsi(outout).replaceAll("\\", "/").replaceAll("×", "✖");
234232

235233
Then("the output should contain", function (this: ICustomWorld, content) {
236-
assert.match(normalizeOutput(this), new RegExp(rescape(content)));
234+
assert.match(
235+
normalizeOutput(expectLastRun(this).stdout),
236+
new RegExp(rescape(content)),
237+
);
238+
});
239+
240+
Then("stderr should contain", function (this: ICustomWorld, content) {
241+
assert.match(
242+
normalizeOutput(expectLastRun(this).stderr),
243+
new RegExp(rescape(content)),
244+
);
237245
});
238246

239247
Then(
240248
"the output should not contain {string}",
241249
function (this: ICustomWorld, content) {
242-
assert.doesNotMatch(normalizeOutput(this), new RegExp(rescape(content)));
250+
assert.doesNotMatch(
251+
normalizeOutput(expectLastRun(this).stdout),
252+
new RegExp(rescape(content)),
253+
);
243254
},
244255
);
245256

‎lib/add-cucumber-preprocessor-plugin.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ import {
1919
TASK_TEST_STEP_STARTED,
2020
TASK_TEST_STEP_FINISHED,
2121
TASK_TEST_CASE_FINISHED,
22+
TASK_FRONTEND_TRACKING_ERROR,
2223
} from "./cypress-task-definitions";
2324

2425
import {
@@ -34,6 +35,7 @@ import {
3435
testStepFinishedHandler,
3536
testCaseFinishedHandler,
3637
OnAfterStep,
38+
frontendTrackingErrorHandler,
3739
} from "./plugin-event-handlers";
3840

3941
import { resolve as origResolve } from "./preprocessor-configuration";
@@ -129,6 +131,10 @@ export async function addCucumberPreprocessorPlugin(
129131
null,
130132
config,
131133
),
134+
[TASK_FRONTEND_TRACKING_ERROR]: frontendTrackingErrorHandler.bind(
135+
null,
136+
config,
137+
),
132138
});
133139

134140
const tags = getTags(config.env);

0 commit comments

Comments
 (0)