Skip to content

Commit 80cd04d

Browse files
trevrosentrevrosen
committed
Land rapid7#4332, test optimization for Cucumber
* Make Cuke run faster on TravisCI
2 parents dcf2317 + 8c0610c commit 80cd04d

File tree

13 files changed

+86
-69
lines changed

13 files changed

+86
-69
lines changed

‎.travis.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
1+
bundler_args: --without development coverage
12
env:
2-
- RAKE_TASK=cucumber
3-
# Commenting out the boot tests due to chronic timeouts.
4-
# - RAKE_TASK=cucumber:boot
5-
- RAKE_TASK=spec SPEC_OPTS="--tag content"
6-
- RAKE_TASK=spec SPEC_OPTS="--tag ~content"
3+
- RAKE_TASKS="cucumber cucumber:boot"
4+
- RAKE_TASKS=spec SPEC_OPTS="--tag content"
5+
- RAKE_TASKS=spec SPEC_OPTS="--tag ~content"
76

87
language: ruby
98
matrix:
109
fast_finish: true
1110
before_install:
11+
- "echo 'gem: --no-ri --no-rdoc' > ~/.gemrc"
1212
- rake --version
1313
- sudo apt-get update -qq
1414
- sudo apt-get install -qq libpcap-dev
@@ -25,7 +25,7 @@ before_script:
2525
- bundle exec rake db:migrate
2626
script:
2727
# fail build if db/schema.rb update is not committed
28-
- git diff --exit-code && bundle exec rake $RAKE_TASK
28+
- git diff --exit-code && bundle exec rake $RAKE_TASKS
2929

3030
rvm:
3131
- '1.9.3'

‎Gemfile

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,14 @@ source 'https://rubygems.org'
33
# spec.add_runtime_dependency '<name>', [<version requirements>]
44
gemspec name: 'metasploit-framework'
55

6+
# separate from test as simplecov is not run on travis-ci
7+
group :coverage do
8+
# code coverage for tests
9+
# any version newer than 0.5.4 gives an Encoding error when trying to read the source files.
10+
# see: https://github.com/colszowka/simplecov/issues/127 (hopefully fixed in 0.8.0)
11+
gem 'simplecov', '0.5.4', :require => false
12+
end
13+
614
group :db do
715
gemspec name: 'metasploit-framework-db'
816
end
@@ -44,10 +52,6 @@ group :test do
4452
# cucumber + automatic database cleaning with database_cleaner
4553
gem 'cucumber-rails', :require => false
4654
gem 'shoulda-matchers'
47-
# code coverage for tests
48-
# any version newer than 0.5.4 gives an Encoding error when trying to read the source files.
49-
# see: https://github.com/colszowka/simplecov/issues/127 (hopefully fixed in 0.8.0)
50-
gem 'simplecov', '0.5.4', :require => false
5155
# Manipulate Time.now in specs
5256
gem 'timecop'
5357
end

‎config/application.rb

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99

1010
Bundler.require(
1111
*Rails.groups(
12+
coverage: [:test],
1213
db: all_environments,
1314
pcap: all_environments
1415
)

‎features/commands/help.feature

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,9 @@
11
Feature: Help command
22

33
Background:
4-
Given I run `msfconsole` interactively
5-
And I wait for stdout to contain "Free Metasploit Pro trial: http://r-7.co/trymsp"
4+
Given I run `msfconsole --defer-module-loads -x help -x exit`
65

76
Scenario: The 'help' command's output
8-
When I type "help"
9-
And I type "exit"
107
Then the output should contain:
118
"""
129
Core Commands

‎features/msfconsole/database_yml.feature

Lines changed: 6 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,7 @@ Feature: `msfconsole` `database.yml`
4848
database: project_metasploit_framework_test
4949
username: project_metasploit_framework_test
5050
"""
51-
When I run `msfconsole --environment test --yaml command_line.yml` interactively
52-
And I wait for stdout to contain "Free Metasploit Pro trial: http://r-7.co/trymsp"
53-
And I type "exit"
51+
When I run `msfconsole --defer-module-loads --environment test --execute-command exit --yaml command_line.yml`
5452
Then the output should contain "command_line_metasploit_framework_test"
5553

5654
Scenario: Without --yaml, MSF_DATABASE_CONFIG wins
@@ -84,9 +82,7 @@ Feature: `msfconsole` `database.yml`
8482
database: project_metasploit_framework_test
8583
username: project_metasploit_framework_test
8684
"""
87-
When I run `msfconsole --environment test` interactively
88-
And I wait for stdout to contain "Free Metasploit Pro trial: http://r-7.co/trymsp"
89-
And I type "exit"
85+
When I run `msfconsole --defer-module-loads --environment test --execute-command exit`
9086
Then the output should contain "environment_metasploit_framework_test"
9187

9288
Scenario: Without --yaml or MSF_DATABASE_CONFIG, ~/.msf4/database.yml wins
@@ -113,9 +109,7 @@ Feature: `msfconsole` `database.yml`
113109
database: project_metasploit_framework_test
114110
username: project_metasploit_framework_test
115111
"""
116-
When I run `msfconsole --environment test` interactively
117-
And I wait for stdout to contain "Free Metasploit Pro trial: http://r-7.co/trymsp"
118-
And I type "exit"
112+
When I run `msfconsole --defer-module-loads --environment test --execute-command exit`
119113
Then the output should contain "user_metasploit_framework_test"
120114

121115
Scenario: Without --yaml, MSF_DATABASE_CONFIG or ~/.msf4/database.yml, project "database.yml" wins
@@ -133,9 +127,7 @@ Feature: `msfconsole` `database.yml`
133127
database: project_metasploit_framework_test
134128
username: project_metasploit_framework_test
135129
"""
136-
When I run `msfconsole --environment test` interactively
137-
And I wait for stdout to contain "Free Metasploit Pro trial: http://r-7.co/trymsp"
138-
And I type "exit"
130+
When I run `msfconsole --defer-module-loads --environment test --execute-command exit`
139131
Then the output should contain "project_metasploit_framework_test"
140132

141133

@@ -148,20 +140,14 @@ Feature: `msfconsole` `database.yml`
148140
And a mocked home directory
149141
And I cd to "../.."
150142
And the project "database.yml" does not exist
151-
When I run `msfconsole --environment test` interactively
152-
And I wait for stdout to contain "Free Metasploit Pro trial: http://r-7.co/trymsp"
153-
And I type "db_status"
154-
And I type "exit"
143+
When I run `msfconsole --defer-module-loads --environment test --execute-command db_status --execute-command exit`
155144
Then the output should not contain "command_line_metasploit_framework_test"
156145
And the output should not contain "environment_metasploit_framework_test"
157146
And the output should not contain "user_metasploit_framework_test"
158147
And the output should not contain "project_metasploit_framework_test"
159148
And the output should contain "[*] postgresql selected, no connection"
160149

161150
Scenario: Starting `msfconsole` with a valid database.yml
162-
Given I run `msfconsole` interactively
163-
And I wait for stdout to contain "Free Metasploit Pro trial: http://r-7.co/trymsp"
164-
When I type "db_status"
165-
And I type "exit"
151+
When I run `msfconsole --defer-module-loads --execute-command db_status --execute-command exit`
166152
Then the output should contain "[*] postgresql connected to metasploit_framework_test"
167153

‎features/support/env.rb

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
# Has to be the first file required so that all other files show coverage information
2-
require 'simplecov'
1+
# @note `require 'simplecov'` is not used here because all features currently use external `msfconsole` process, so only
2+
# that child process needs to load 'simplecov'.
33

44
# IMPORTANT: This file is generated by cucumber-rails - edit at your own peril.
55
# It is recommended to regenerate this file in the future when you upgrade to a

‎features/support/hooks.rb

Lines changed: 19 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,23 +4,26 @@
44
@aruba_timeout_seconds = 8.minutes
55
end
66

7-
Before do |scenario|
8-
command_name = case scenario
9-
when Cucumber::Ast::Scenario, Cucumber::Ast::ScenarioOutline
10-
"#{scenario.feature.title} #{scenario.name}"
11-
when Cucumber::Ast::OutlineTable::ExampleRow
12-
scenario_outline = scenario.scenario_outline
7+
# don't setup child processes to load simplecov_setup.rb if simplecov isn't installed
8+
unless Bundler.settings.without.include?(:coverage)
9+
Before do |scenario|
10+
command_name = case scenario
11+
when Cucumber::Ast::Scenario, Cucumber::Ast::ScenarioOutline
12+
"#{scenario.feature.title} #{scenario.name}"
13+
when Cucumber::Ast::OutlineTable::ExampleRow
14+
scenario_outline = scenario.scenario_outline
1315

14-
"#{scenario_outline.feature.title} #{scenario_outline.name} #{scenario.name}"
15-
else
16-
raise TypeError, "Don't know how to extract command name from #{scenario.class}"
17-
end
16+
"#{scenario_outline.feature.title} #{scenario_outline.name} #{scenario.name}"
17+
else
18+
raise TypeError, "Don't know how to extract command name from #{scenario.class}"
19+
end
1820

19-
# Used in simplecov_setup so that each scenario has a different name and their coverage results are merged instead
20-
# of overwriting each other as 'Cucumber Features'
21-
set_env('SIMPLECOV_COMMAND_NAME', command_name)
21+
# Used in simplecov_setup so that each scenario has a different name and their coverage results are merged instead
22+
# of overwriting each other as 'Cucumber Features'
23+
set_env('SIMPLECOV_COMMAND_NAME', command_name)
2224

23-
simplecov_setup_pathname = Pathname.new(__FILE__).expand_path.parent.join('simplecov_setup')
24-
# set environment variable so child processes will merge their coverage data with parent process's coverage data.
25-
set_env('RUBYOPT', "-r#{simplecov_setup_pathname} #{ENV['RUBYOPT']}")
25+
simplecov_setup_pathname = Pathname.new(__FILE__).expand_path.parent.join('simplecov_setup')
26+
# set environment variable so child processes will merge their coverage data with parent process's coverage data.
27+
set_env('RUBYOPT', "#{ENV['RUBYOPT']} -r#{simplecov_setup_pathname}")
28+
end
2629
end

‎features/support/simplecov_setup.rb

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,16 @@
11
# @note this file is loaded in env.rb to setup simplecov using RUBYOPTs for child processes
22

3-
require 'simplecov'
3+
simplecov_command_name = ENV['SIMPLECOV_COMMAND_NAME']
44

5-
require 'pathname'
5+
# will not be set if hook does not run because `bundle install --without coverage`
6+
if simplecov_command_name
7+
require 'simplecov'
68

7-
root = Pathname(__FILE__).expand_path.parent.parent.parent
9+
require 'pathname'
810

9-
SimpleCov.command_name(ENV['SIMPLECOV_COMMAND_NAME'])
10-
SimpleCov.root(root)
11-
load root.join('.simplecov')
11+
root = Pathname(__FILE__).expand_path.parent.parent.parent
12+
13+
SimpleCov.command_name(simplecov_command_name)
14+
SimpleCov.root(root)
15+
load root.join('.simplecov')
16+
end

‎lib/metasploit/framework/command/console.rb

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ def driver_options
6161
driver_options['DatabaseEnv'] = options.environment
6262
driver_options['DatabaseMigrationPaths'] = options.database.migrations_paths
6363
driver_options['DatabaseYAML'] = options.database.config
64+
driver_options['DeferModuleLoads'] = options.modules.defer_loads
6465
driver_options['Defanged'] = options.console.defanged
6566
driver_options['DisableBanner'] = options.console.quiet
6667
driver_options['DisableDatabase'] = options.database.disable

‎lib/metasploit/framework/parsed_options/base.rb

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ def options
8585
options.framework.config = nil
8686

8787
options.modules = ActiveSupport::OrderedOptions.new
88+
options.modules.defer_loads = false
8889
options.modules.path = nil
8990

9091
@options = options
@@ -155,6 +156,13 @@ def option_parser
155156
option_parser.separator ''
156157
option_parser.separator 'Module options'
157158

159+
option_parser.on(
160+
'--defer-module-loads',
161+
'Defer module loading unless explicitly asked.'
162+
) do
163+
options.modules.defer_loads = true
164+
end
165+
158166
option_parser.on(
159167
'-m',
160168
'--module-path DIRECTORY',

0 commit comments

Comments
 (0)