Skip to content

Commit 43825f9

Browse files
avmnu-sngavmnu-sng
authored
[Fix #41] Ignore Duplicate Examples in Reports (#42)
Currently, the RSpec Tracer stops generating reports in case of duplicate examples. Now, allowing it to continue with the reports by ignoring these duplicate examples. The HTML report provides the list of such examples. Also, the default exit code is `1`, which can be controlled using the environment variable `RSPEC_TRACER_FAIL_ON_DUPLICATES`.
1 parent 18099b4 commit 43825f9

File tree

17 files changed

+390
-104
lines changed

17 files changed

+390
-104
lines changed

‎.rubocop.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@ Metrics/BlockLength:
4444
Metrics/ClassLength:
4545
Max: 300
4646

47+
Metrics/CyclomaticComplexity:
48+
Max: 10
49+
4750
Metrics/MethodLength:
4851
Max: 25
4952

‎README.md‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ recommended to use **simplecov >= 0.12.0**. To use RSpec Tracer **cache on CI**,
2828
need to have an **S3 bucket** and **[AWS CLI](https://aws.amazon.com/cli/)**
2929
installed.
3030

31-
### You should take some time and go through the **[document](./RSPEC_TRACER.md)** describing the **intention** and implementation details of **managing dependency**, **managing flaky tests**, **skipping tests**, and **caching on CI**.
31+
> You should take some time and go through the **[document](./RSPEC_TRACER.md)** describing the **intention** and implementation details of **managing dependency**, **managing flaky tests**, **skipping tests**, and **caching on CI**.
3232
3333
## Table of Contents
3434

@@ -38,7 +38,7 @@ installed.
3838
* [Advanced Configuration](#advanced-configuration)
3939
* [Filters](#filters)
4040
* [Environment Variables](#environment-variables)
41-
* [When Should You Not Use RSpec Tracer](#when-should-you-not-use-rspec-tracer)
41+
* [Duplicate Examples](#duplicate-examples)
4242

4343
## Demo
4444

@@ -62,6 +62,12 @@ These reports provide basic test information:
6262

6363
![](./readme_files/examples_report_next_run.png)
6464

65+
### Duplicate Examples Report
66+
67+
These reports provide duplicate tests information.
68+
69+
![](./readme_files/duplicate_examples_report.png)
70+
6571
### Flaky Examples Report
6672

6773
These reports provide flaky tests information. Assuming **the following two tests
@@ -307,6 +313,9 @@ development environment. You can install [localstack](https://github.com/localst
307313
and [awscli-local](https://github.com/localstack/awscli-local) and then invoke the
308314
rake tasks with `LOCAL_AWS=true`.
309315
316+
- **`RSPEC_TRACER_FAIL_ON_DUPLICATES (default: true)`:** By default, RSpec Tracer
317+
exits with one if there are [duplicate examples](#duplicate-examples).
318+
310319
- **`RSPEC_TRACER_NO_SKIP (default: false)`:** Use this environment variables to
311320
not skip any tests. Note that it will continue to maintain cache files and generate
312321
reports.
@@ -333,7 +342,7 @@ specific test suites and not merge them.
333342
TEST_SUITE_ID=2 bundle exec rspec spec/helpers
334343
```
335344
336-
## When Should You Not Use RSpec Tracer
345+
## Duplicate Examples
337346
338347
To uniquely identify the examples is one of the requirements for the correctness
339348
of the RSpec Tracer. Sometimes, it would not be possible to do so depending upon
@@ -423,17 +432,12 @@ Calculator
423432
```
424433

425434
In this scenario, RSpec Tracer cannot determine the `Calculator#add` and
426-
`Calculator#sub` group examples. Also, it will ask you not to use the gem unless
427-
you have made some changes to your spec files.
435+
`Calculator#sub` group examples.
428436

429437
```
430438
================================================================================
431-
IMPORTANT NOTICE -- DO NOT USE RSPEC TRACER
432-
================================================================================
433-
It would be best to make changes so that the RSpec tracer can uniquely
434-
identify all the examples, and then you can enable the RSpec tracer back.
439+
IMPORTANT NOTICE -- RSPEC TRACER COULD NOT IDENTIFY SOME EXAMPLES UNIQUELY
435440
================================================================================
436-
437441
RSpec tracer could not uniquely identify the following 10 examples:
438442
- Example ID: eabd51a899db4f64d5839afe96004f03 (5 examples)
439443
* Calculator#add (spec/calculator_spec.rb:13)

‎features/data/calculator_2_app/all_examples.json‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
{
2+
"1be34ddaa19469923b1a2c6798a5d15a": {
3+
"example_group": "RSpec::ExampleGroups::Calculator::Mul",
4+
"description": "multiplies 1 and 2 to -2",
5+
"full_description": "Calculator#mul multiplies 1 and 2 to -2",
6+
"file_name": "/spec/calculator_spec.rb",
7+
"line_number": 38,
8+
"rerun_file_name": "/spec/calculator_spec.rb",
9+
"rerun_line_number": 38,
10+
"example_id": "1be34ddaa19469923b1a2c6798a5d15a"
11+
},
12+
"743241f052ac078df639300bf9e7dc90": {
13+
"example_group": "RSpec::ExampleGroups::Calculator::Mul",
14+
"description": "multiplies 10 and 0 to 0",
15+
"full_description": "Calculator#mul multiplies 10 and 0 to 0",
16+
"file_name": "/spec/calculator_spec.rb",
17+
"line_number": 38,
18+
"rerun_file_name": "/spec/calculator_spec.rb",
19+
"rerun_line_number": 38,
20+
"example_id": "743241f052ac078df639300bf9e7dc90"
21+
},
22+
"399a3130aaca80408020db0d3d2a1571": {
23+
"example_group": "RSpec::ExampleGroups::Calculator::Mul",
24+
"description": "multiplies 5 and 7 to 35",
25+
"full_description": "Calculator#mul multiplies 5 and 7 to 35",
26+
"file_name": "/spec/calculator_spec.rb",
27+
"line_number": 38,
28+
"rerun_file_name": "/spec/calculator_spec.rb",
29+
"rerun_line_number": 38,
30+
"example_id": "399a3130aaca80408020db0d3d2a1571"
31+
},
32+
"3e2f78d04a432fd57e6e173732dfc665": {
33+
"example_group": "RSpec::ExampleGroups::Calculator::Mul",
34+
"description": "multiplies -1 and -8 to 8",
35+
"full_description": "Calculator#mul multiplies -1 and -8 to 8",
36+
"file_name": "/spec/calculator_spec.rb",
37+
"line_number": 38,
38+
"rerun_file_name": "/spec/calculator_spec.rb",
39+
"rerun_line_number": 38,
40+
"example_id": "3e2f78d04a432fd57e6e173732dfc665"
41+
},
42+
"6dc2c97f1c877ff305a42c5a30454f25": {
43+
"example_group": "RSpec::ExampleGroups::Calculator::Mul",
44+
"description": "multiplies 10 and 10 to 100",
45+
"full_description": "Calculator#mul multiplies 10 and 10 to 100",
46+
"file_name": "/spec/calculator_spec.rb",
47+
"line_number": 38,
48+
"rerun_file_name": "/spec/calculator_spec.rb",
49+
"rerun_line_number": 38,
50+
"example_id": "6dc2c97f1c877ff305a42c5a30454f25"
51+
}
52+
}

‎features/data/calculator_2_app/coverage.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
{
2+
"app/calculator.rb": [
3+
null,
4+
null,
5+
1,
6+
1,
7+
null,
8+
6,
9+
6,
10+
6,
11+
null
12+
]
13+
}

‎features/data/calculator_2_app/dependency.json‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"399a3130aaca80408020db0d3d2a1571": [
3+
"/spec/calculator_spec.rb",
4+
"/app/calculator.rb",
5+
"/spec/spec_helper.rb"
6+
],
7+
"743241f052ac078df639300bf9e7dc90": [
8+
"/spec/calculator_spec.rb",
9+
"/app/calculator.rb",
10+
"/spec/spec_helper.rb"
11+
],
12+
"3e2f78d04a432fd57e6e173732dfc665": [
13+
"/spec/calculator_spec.rb",
14+
"/app/calculator.rb",
15+
"/spec/spec_helper.rb"
16+
],
17+
"6dc2c97f1c877ff305a42c5a30454f25": [
18+
"/spec/calculator_spec.rb",
19+
"/app/calculator.rb",
20+
"/spec/spec_helper.rb"
21+
],
22+
"1be34ddaa19469923b1a2c6798a5d15a": [
23+
"/spec/calculator_spec.rb",
24+
"/app/calculator.rb",
25+
"/spec/spec_helper.rb"
26+
]
27+
}

‎features/data/calculator_2_app/reverse_dependency.json‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{
2+
"/app/calculator.rb": {
3+
"example_count": 5,
4+
"examples": {
5+
"/spec/calculator_spec.rb": 5
6+
}
7+
},
8+
"/spec/calculator_spec.rb": {
9+
"example_count": 5,
10+
"examples": {
11+
"/spec/calculator_spec.rb": 5
12+
}
13+
},
14+
"/spec/spec_helper.rb": {
15+
"example_count": 5,
16+
"examples": {
17+
"/spec/calculator_spec.rb": 5
18+
}
19+
}
20+
}

‎features/ruby_app_with_incorrect_analysis.feature‎

Lines changed: 85 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,93 @@ Feature: Ruby App with Incorrect Analysis
2121
multiplies 10 and 10 to 100 (No cache)
2222
multiplies 10 and 0 to 0 (No cache)
2323
"""
24-
And The RSpecTracer should forbid using the tool
24+
And The RSpecTracer should print "example at ./spec/calculator_spec.rb:13 (No cache)" example 5 times
25+
And The RSpecTracer should print "performs subtraction (No cache)" example 5 times
26+
And The RSpecTracer report should have been generated
27+
And The last run report should have correct details
28+
"""
29+
{
30+
"run_id": "35194a37e68446e9d6960c46e717fd44",
31+
"actual_count": 15,
32+
"example_count": 15,
33+
"skipped_examples": 0,
34+
"failed_examples": 1,
35+
"pending_examples": 0
36+
}
37+
"""
38+
And The all examples report should have correct details
39+
And The all files report should have correct details
40+
| file_name | file_digest |
41+
| /spec/spec_helper.rb | 4f20a4832e5ca0e26d1ffe5a4a766a1f |
42+
| /spec/calculator_spec.rb | 6b72931a56345d552dd5ac1fd60909f3 |
43+
| /app/calculator.rb | a3d17fca6ecabbbe108281e7847a24ee |
44+
And The failed example report should have correct details
45+
And The pending example report should have correct details
46+
And The dependency report should have correct details
47+
And The reverse dependency report should have correct details
48+
And The JSON coverage report should have been generated for "RSpec"
49+
And The coverage percent stat is "5 / 5 LOC (100.0%) covered"
50+
And The JSON coverage report should have correct coverage for "RSpec"
51+
And The RSpecTracer should print the duplicate examples report
52+
"""
53+
================================================================================
54+
IMPORTANT NOTICE -- RSPEC TRACER COULD NOT IDENTIFY SOME EXAMPLES UNIQUELY
55+
================================================================================
56+
RSpec tracer could not uniquely identify the following 10 examples:
57+
- Example ID: eabd51a899db4f64d5839afe96004f03 (5 examples)
58+
* Calculator#add (spec/calculator_spec.rb:13)
59+
* Calculator#add (spec/calculator_spec.rb:13)
60+
* Calculator#add (spec/calculator_spec.rb:13)
61+
* Calculator#add (spec/calculator_spec.rb:13)
62+
* Calculator#add (spec/calculator_spec.rb:13)
63+
- Example ID: 72171b502c5a42b9aa133f165cf09ec2 (5 examples)
64+
* Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
65+
* Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
66+
* Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
67+
* Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
68+
* Calculator#sub performs subtraction (spec/calculator_spec.rb:24)
69+
"""
70+
When I run `bundle exec rspec spec`
71+
Then The RSpecTracer should print the information
72+
"""
73+
Started RSpec tracer
74+
RSpec tracer is running 11 examples (actual: 15, skipped: 4)
75+
example at ./spec/calculator_spec.rb:13 (No cache)
76+
performs subtraction (No cache)
77+
multiplies 1 and 2 to -2 (Failed previously) (FAILED - 1)
78+
"""
79+
And The RSpecTracer should print "example at ./spec/calculator_spec.rb:13 (No cache)" example 5 times
80+
And The RSpecTracer should print "performs subtraction (No cache)" example 5 times
81+
And The RSpecTracer report should have been generated
82+
And The last run report should have correct details
83+
"""
84+
{
85+
"run_id": "35194a37e68446e9d6960c46e717fd44",
86+
"actual_count": 15,
87+
"example_count": 11,
88+
"skipped_examples": 4,
89+
"failed_examples": 1,
90+
"pending_examples": 0
91+
}
92+
"""
93+
And The all examples report should have correct details
94+
And The all files report should have correct details
95+
| file_name | file_digest |
96+
| /spec/spec_helper.rb | 4f20a4832e5ca0e26d1ffe5a4a766a1f |
97+
| /spec/calculator_spec.rb | 6b72931a56345d552dd5ac1fd60909f3 |
98+
| /app/calculator.rb | a3d17fca6ecabbbe108281e7847a24ee |
99+
And The failed example report should have correct details
100+
And The pending example report should have correct details
101+
And The dependency report should have correct details
102+
And The reverse dependency report should have correct details
103+
And The JSON coverage report should have been generated for "RSpec"
104+
And The coverage percent stat is "5 / 5 LOC (100.0%) covered"
105+
And The JSON coverage report should have correct coverage for "RSpec"
25106
And The RSpecTracer should print the duplicate examples report
26107
"""
108+
================================================================================
109+
IMPORTANT NOTICE -- RSPEC TRACER COULD NOT IDENTIFY SOME EXAMPLES UNIQUELY
110+
================================================================================
27111
RSpec tracer could not uniquely identify the following 10 examples:
28112
- Example ID: eabd51a899db4f64d5839afe96004f03 (5 examples)
29113
* Calculator#add (spec/calculator_spec.rb:13)

‎features/step_definitions/project_steps.rb‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
rails_app: '6654a84c672a717904112cef7503d7a1',
1010
ruby_app: '63df6c782675a201fbef23140bd868e2',
1111
calculator_app: 'ac50ff82ef0e8c97f7142ae07483d81d',
12+
calculator_2_app: '35194a37e68446e9d6960c46e717fd44',
1213
calculator_3_app: 'ac50ff82ef0e8c97f7142ae07483d81d'
1314
}[@project.to_sym]
1415

‎features/step_definitions/report_steps.rb‎

Lines changed: 13 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,12 @@
1515
expect(output & expected).to contain_exactly(*expected)
1616
end
1717

18+
Then('The RSpecTracer should print {string} example {int} times') do |example, times|
19+
output = last_command_started.output.lines.map(&:strip).reject(&:empty?)
20+
21+
expect(output.count(example)).to eq(times)
22+
end
23+
1824
Then('The SimpleCov at_exit hook should have executed at the right time') do
1925
output = last_command_started.output.lines.map(&:strip).reject(&:empty?)
2026
idx = output.index('SimpleCov will now generate coverage report (<3 RSpec tracer)')
@@ -33,19 +39,6 @@
3339
expect(output).to eq(expected)
3440
end
3541

36-
Then('The RSpecTracer should forbid using the tool') do
37-
expected = [
38-
'================================================================================',
39-
' IMPORTANT NOTICE -- DO NOT USE RSPEC TRACER',
40-
'================================================================================',
41-
' It would be best to make changes so that the RSpec tracer can uniquely',
42-
' identify all the examples, and then you can enable the RSpec tracer back.',
43-
'================================================================================'
44-
].join("\n")
45-
46-
expect(last_command_started.output).to include(expected)
47-
end
48-
4942
Then('The RSpecTracer report should have been generated') do
5043
steps %(
5144
Then a directory named "#{@cache_dir}" should exist
@@ -133,6 +126,8 @@
133126
else
134127
['b5963ecab8d95c1024a46117fce4e907']
135128
end
129+
when 'calculator_2_app'
130+
['1be34ddaa19469923b1a2c6798a5d15a']
136131
end
137132

138133
expect(report).to eq(example)
@@ -143,7 +138,11 @@
143138
cd('.') do
144139
report = JSON.parse(File.read("#{@cache_dir}/#{@run_id}/pending_examples.json"))
145140

146-
expect(report).to eq(['94cd4d0e1d9ef63237421fe02085eb9a'])
141+
if @project == 'calculator_2_app'
142+
expect(report).to eq([])
143+
else
144+
expect(report).to eq(['94cd4d0e1d9ef63237421fe02085eb9a'])
145+
end
147146
end
148147
end
149148

‎lib/rspec_tracer.rb‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,16 +67,18 @@ def filter_examples
6767
end
6868
end
6969

70+
runner.deregister_duplicate_examples
71+
7072
[to_run, groups.to_a]
7173
end
7274
# rubocop:enable Metrics/AbcSize
7375

7476
def at_exit_behavior
7577
return unless RSpecTracer.pid == Process.pid && RSpecTracer.running
7678

77-
::Kernel.exit(1) if runner.incorrect_analysis?
78-
7979
run_exit_tasks
80+
81+
::Kernel.exit(1) if runner.non_zero_exit_code?
8082
ensure
8183
RSpecTracer.running = false
8284
end

0 commit comments

Comments
 (0)