|
| 1 | +--- |
| 2 | +layout: default |
| 3 | +title: Test explorer |
| 4 | +parent: VS Code extension |
| 5 | +--- |
| 6 | + |
| 7 | +# Test explorer |
| 8 | + |
| 9 | +{: .important } |
| 10 | +The new test explorer implementation is currently being rolled out to users! You can adopt it early by toggling this |
| 11 | +feature flag in your user or workspace settings<br> |
| 12 | +"rubyLsp.featureFlags": { "fullTestDiscovery": true } |
| 13 | + |
| 14 | +The Ruby LSP implements VS Code's [test explorer](https://code.visualstudio.com/docs/debugtest/testing), which allows |
| 15 | +users to execute the tests defined in their codebase in 4 modes directly from inside the editor: |
| 16 | + |
| 17 | +- Run (default mode): runs the selected tests and displays results in the test results panel |
| 18 | +- Run in terminal: runs the selected tests in a terminal set up by the Ruby LSP |
| 19 | +- Debug: starts an interactive debugging session for the selected tests |
| 20 | +- Coverage: runs tests in coverage mode and shows results inside the editor |
| 21 | + |
| 22 | + |
| 23 | + |
| 24 | +## Design |
| 25 | + |
| 26 | +Our design is based on addressing 2 main goals: |
| 27 | + |
| 28 | +1. Supporting Ruby's diverse test frameworks without the need for extra editor extensions |
| 29 | +2. Ensuring the solution is performant enough for large scale applications |
| 30 | + |
| 31 | +With these in mind, the Ruby LSP populates the test explorer panel through static analysis. Loading every single test |
| 32 | +into memory to perform runtime introspection as a discovery mechanism would not satisfy our performance goal. Tests |
| 33 | +are discovered for the entire codebase automatically when: |
| 34 | + |
| 35 | +- the user clicks one of the test related [code lenses](index#code-lens) |
| 36 | +- the user expands the explorer |
| 37 | + |
| 38 | +Support for different frameworks can be provided via our [add-on API](add-ons), both for discovering tests and defining how to execute them. Any framework contribution made via add-ons is automatically integrated with all modes |
| 39 | +of execution. By default, the Ruby LSP supports Minitest and Test Unit. When working on Rails applications, the Rails |
| 40 | +add-on is automatically included to support the declarative syntax included by `ActiveSupport::TestCase`. |
| 41 | + |
| 42 | +{: .important } |
| 43 | +There is limited support to using multiple test frameworks in the same codebase. This use case is pretty uncommon |
| 44 | +and we will not make further investments into supporting it, in line with our design principle of [favoring common |
| 45 | +setups](design-and-roadmap#favoring-common-development-setups) |
| 46 | + |
| 47 | +### Dynamically defined tests |
| 48 | + |
| 49 | +There is limited support for tests defined via meta-programming. Initially, they will not be present in the test |
| 50 | +explorer (as they often cannot be detected through static analysis). However, running a test file that includes |
| 51 | +dynamically defined tests will automatically populate the explorer with those tests, including the results of the |
| 52 | +execution. |
| 53 | + |
| 54 | +```ruby |
| 55 | +class MyTest < Minitest::Spec |
| 56 | + # These are detected automatically |
| 57 | + describe "something" do |
| 58 | + it "does a useful thing" do |
| 59 | + end |
| 60 | + end |
| 61 | + |
| 62 | + # Dynamically defined tests like these are only discovered while running the entire file |
| 63 | + [:first, :second, :third].each do |name| |
| 64 | + it "does the #{name} well" do |
| 65 | + end |
| 66 | + end |
| 67 | +end |
| 68 | +``` |
| 69 | + |
| 70 | +### Tests that accept external parameters |
| 71 | + |
| 72 | +In Ruby, you can write tests that accept external parameters, like environment variables. |
| 73 | + |
| 74 | +```ruby |
| 75 | +class MyTest < Minitest::Test |
| 76 | + # Using instance variable as an external argument |
| 77 | + if ENV["INCLUDE_SLOW_TESTS"] |
| 78 | + def test_slow_operation |
| 79 | + end |
| 80 | + end |
| 81 | + |
| 82 | + # Using command line arguments to gate tests |
| 83 | + if ARGV.include?("--integration-tests") |
| 84 | + def test_integration |
| 85 | + end |
| 86 | + end |
| 87 | + |
| 88 | + def test_other_things |
| 89 | + end |
| 90 | +end |
| 91 | +``` |
| 92 | + |
| 93 | +Automatically detecting what type of external argument is required for each test is not trivial. Additionally, VS |
| 94 | +Code's test explorer doesn't have support for arguments when running tests out of the box and neither do its test |
| 95 | +items accept metadata. This scenario will not be supported by the Ruby LSP. |
| 96 | + |
| 97 | +## Customization |
| 98 | + |
| 99 | +When tests are running through any execution mode, we set the `RUBY_LSP_TEST_RUNNER` environment variable to allow |
| 100 | +users to customize behavior of their test suite if needed. |
| 101 | + |
| 102 | +{: .important } |
| 103 | +The Ruby LSP uses a custom test reporter to be able to communicate between extension and server. Some gems that modify |
| 104 | +reporters may break this integration. The `RUBY_LSP_TEST_RUNNER` variable can be used to turn off these gems only when |
| 105 | +running under the Ruby LSP's integrations. |
| 106 | + |
| 107 | +{: .important } |
| 108 | +Using coverage mode **does not require any extra dependencies or configuration** for collecting the coverage data. This is done automatically by the Ruby LSP through Ruby's built-in coverage API. |
| 109 | + |
| 110 | +For example |
| 111 | + |
| 112 | +```ruby |
| 113 | +# test/test_helper.rb |
| 114 | + |
| 115 | +unless ENV["RUBY_LSP_TEST_RUNNER"] |
| 116 | + # Minitest reporters cannot be used when running through the Ruby LSP integrations as it breaks our custom reporter |
| 117 | + |
| 118 | + require "minitest/reporters" |
| 119 | + Minitest::Reporters.use!(...) |
| 120 | +end |
| 121 | +``` |
| 122 | + |
| 123 | +Users can also differentiate between the mode of execution, which is the value of the `RUBY_LSP_TEST_RUNNER` variable: |
| 124 | + |
| 125 | +```ruby |
| 126 | +# test/test_helper.rb |
| 127 | + |
| 128 | +case ENV["RUBY_LSP_TEST_RUNNER"] |
| 129 | +when "run" |
| 130 | + # Do something when using run or run in terminal modes |
| 131 | +when "debug" |
| 132 | + # Do something when using debug mode |
| 133 | +when "coverage" |
| 134 | + # Do something when using coverage mode |
| 135 | +else |
| 136 | + # Do something when running outside of the context of the Ruby LSP integration |
| 137 | +end |
| 138 | +``` |
| 139 | + |
| 140 | +## Other editors |
| 141 | + |
| 142 | +The test explorer functionality is not yet standardized as part of the |
| 143 | +[language server specification](https://microsoft.github.io/language-server-protocol/specification), which means that |
| 144 | +it cannot be used by other editors without custom extension code to integrate all of the pieces together. |
| 145 | + |
| 146 | +As most of the implementation is on server side, if any editor supports similar UI elements and editor-side APIs |
| 147 | +(either directly or through plugins), it can integrate this feature as well. Below are the custom request |
| 148 | +specifications. |
| 149 | + |
| 150 | +### Discover tests |
| 151 | + |
| 152 | +This request is sent by the client to discover which test items exist for a given text document URI. |
| 153 | + |
| 154 | +Server capability: `capabilities.experimental.full_test_discovery` |
| 155 | + |
| 156 | +Method: `rubyLsp/discoverTests` |
| 157 | + |
| 158 | +Params: |
| 159 | + |
| 160 | +```typescript |
| 161 | +interface DiscoverTestParams { |
| 162 | + textDocument: { |
| 163 | + uri: string; |
| 164 | + }; |
| 165 | +} |
| 166 | +``` |
| 167 | + |
| 168 | +Response: |
| 169 | + |
| 170 | +```typescript |
| 171 | +// Matches vscode.TestItem with some minor modifications |
| 172 | +interface TestItem { |
| 173 | + id: string; |
| 174 | + label: string; |
| 175 | + uri: string; |
| 176 | + range: { start: { line: number; character: number }, end: { line: number; character: number }}; |
| 177 | + tags: string[]; |
| 178 | + children: TestItem[]; |
| 179 | +} |
| 180 | + |
| 181 | +type Response = TestItem[]; |
| 182 | +``` |
| 183 | + |
| 184 | +### Resolve test commands |
| 185 | + |
| 186 | +This request is sent by the client for the server to determine the minimum number of commands required to execute a |
| 187 | +given hierarchy of tests. For example, if we execute a test group (class) inside of the bar_test.rb file and 3 |
| 188 | +examples inside of the `foo_test.rb` file, the minimum required commands to execute them may look like this: |
| 189 | + |
| 190 | +```ruby |
| 191 | +[ |
| 192 | + "bin/rails test test/foo_test.rb:13:25:40", |
| 193 | + "bin/rails test test/bar_test.rb --name \"/^BarTest::NestedTest(#|::)/\"" |
| 194 | +] |
| 195 | +``` |
| 196 | + |
| 197 | +Server capability: `capabilities.experimental.full_test_discovery` |
| 198 | + |
| 199 | +Method: `rubyLsp/resolveTestCommands` |
| 200 | + |
| 201 | +Params: |
| 202 | + |
| 203 | +```typescript |
| 204 | +type Params = TestItem[]; |
| 205 | +``` |
| 206 | + |
| 207 | +Response: |
| 208 | + |
| 209 | +```typescript |
| 210 | +interface ResolveTestCommandsResult { |
| 211 | + // The array of commands required to execute the tests |
| 212 | + commands: string[]; |
| 213 | + |
| 214 | + // An optional array of custom LSP test reporters. Used to stream test results to the client side using JSON RPC |
| 215 | + // messages |
| 216 | + reporterPaths?: string[]; |
| 217 | +} |
| 218 | +``` |
0 commit comments