The Ecstasy CLI

[thegridman]

This discussion is to work through some ideas and proposals for the Ecstasy command line tools. Each of these tools has its own set of command line options to control its behavior. The reason for starting this discussion is that we will need a way for developers to execute tests from the command line and there are a few options available.

Ecstasy currently has three command line tools:

• xtc - to compile Ecstacy code
• xec - to execute compiled code
• xam - disassemble Ecstacy code

Executing Tests

Tests are executed by the Unit engine module. When executing tests there are a number of options that the developer can supply to control how the tests execute.

Test Execution Options

The list below shows some options that could be used to control test execution. This list is by no means a definite set of option names, it is mainly as an example of the sort of options that could be required to give flexibility in executing tests.

Options to Control Which Tests are Executed

This is where we want to only run specific tests, for example only those related to the specific area of code we're currently working on.

• --group
  Only run tests in a specific group. A test's group is specified in code as one of the properties of the @Test mixin. For example: --group Slow only run tests in the Slow group.

• --package
  Only run tests in a package name that matches a regex. This option could be repeated to run tests in multiple packages. For example: --package myapp.webserver this would only run tests in the myapp.webserver package.

• --class
  Only run tests in a class name that matches a regex. This option could be repeated to run tests in multiple classes. For example: --class myapp.webserver.CustomerTests this would only run tests in the myapp.webserver.CustomerTests class.

• --test
  Only run tests where the method name matches a regex and would probably be combined with --class. This option could be repeated to run tests in multiple tests .For example: --test shouldGetCustomers this would only run the shouldGetCustomers test.

Options to Exclude Specific Tests from Execution

This is where we want to skip tests because they are temporarily broken or for some other reason.

• --exclude-package
  Run all tests but do not run tests in a package name that matches a regex. This option could be repeated to not run tests in multiple packages. For example: --exclude-package myapp.webserver this would not run tests in the myapp.webserver package.

• --exclude-class
  Run all tests but do not run tests in a class name that matches a regex. This is where we want to skip tests because they are temporarily broken or for some other reason. This option could be repeated to not run tests in multiple classes. For example: --exclude-class myapp.webserver.CustomerTests this would not run tests in the myapp.webserver.CustomerTests class.

• --exclude-test
  Only run tests where the method name matches a regex. This is where we only want to skip one or more individual tests in the whole suite. This option could be repeated to not run multiple tests .For example: --exclude-test shouldGetCustomers this would only run the shouldGetCustomers test.

Other Options

There are probably a few other options that could be added. For example:

• --repeat
  Run the tests a number of times. This is useful when investigating intermittent failures where a test can be run a few hundred times. This would normally be combined with options like -c or -t to run specific tests. For example: --repeat 100 run the test 100 times

There may also be options to control how tests are reported, whether to run tests in parallel, whether to stop execution at the first failure, etc.

Command Line Tool Choices

The choices for how we execute tests are:

1. Use the xec tool and add more options like those discussed above to control test execution
2. Write a new xtest command line tool
3. Do something else...

1. Use the xec Tool

The current xec tool is used to execute an Ecstasy programme from the command line. Executing tests is obviously a very similar process, instead of running a module's run() method (or other method using the -m option) for executing tests we would run the XUnit test engine on the module.

Advantages

The tool already exists and runs Ecstasy code.

Disadvantages

There are some options for xec that will not apply when running tests (e.g. -m) and there are a large number of options discussed above that only apply when running tests.

2. Write a New xtest Tool

A new xtest command line tool could be written. This can re-use some of the xec code and support all the new test options.

Advantages

It gets around the problem of mixing normal execution options and test options in one executable

Disadvantages

We are introducing yet another command line tool.

3. Something Else...

The final option is to do something different. We could follow the example of other languages and CLI tools and just have a single command line tool. Instead of different tools we then have different sub-commands. For example, Go has a single tool called go that is run like this: go <command> [options]. Ecstasy could just have a single xtc executable that follows this same pattern.

xtc build

xtc run

xtc test

etc...

Advantages

It gets around the problem of mixing normal execution options and test options
It is easier to extend than adding multiple additions command line tools

Disadvantages

Running a tool requires a bit more typing, e.g. xtc run instead of xec

[ggleyzer]

My biggest concern with options 2) and 3) is that they are geared specifically toward "us" (developers of Ecstasy libraries and tools), rather than "them" - developers of xqiz.it-hosted web applications - our main and most important audience. I don't yet have a clear picture how would "they" use our testing tools, but it's quite clear that it wouldn't be neither of the things listed above, but would have to be some kind of adapter to a well-defined testing API driven by the hosting UI. As such, I would guess that approach 1) gives us a bit of an incentive to provide a crisp and useful API to be used by both us and them

[thegridman]

Right, this is probably where there is some disconnect or misunderstanding then. Are you saying they "customers" who write Ecstasy applications in their IDE will only execute that code in the hosted xqiz.it service? That's a bit different to the developer workflow I am used to.

[ggleyzer]

Yes; consider what you need to do as a "standalone" app developer to use a database. Take a look at WelcomeTest lines 16-22. You need to know the location of the database, provide a directory for the generated code artifacts and manually create the connection.

However, If you are using our hosted environment (welcome), all you do is:

@Inject db.WelcomeSchema schema;

If you're trying to create a WebApp manually, the additional complexity that comes for "no cost" in the hosted environment is even higher. In the range of "hard" to "unattainable"

[cliffclick]

Somewhere, buried in the gradle someplace, we're gonna need tests to test our own tools - as opposed to the XTC version of JUnit.
No customers involved, except ourselves directly.

[cpurdy]

Discussed today: Retain a compile command (perhaps xcc) and a run command (xec); introduce an "umbrella command" (perhaps xtc), a la the Go example.